Wiki source code of Architecture

Last modified by Mikko Heikkinen on 2024/02/02 14:08
Show last authors
1 Enterprise architecture (//operating model// for leadership) exists in Finnish only.
2
3 Julkisen hallinnon kokonaisarkkitehtuuri (JHKA) on rakenne, jonka avulla koordinoidaan ja kehitetään hallinnon organisaatioiden ja palveluiden välistä yhteentoimivuutta. Kokonaisarkkitehtuuri tukee johtamista, strategian toimeenpanoa, toiminnan ja palvelujen jatkuvaa kehittämistä, muutosten ja monimutkaisuuden hallintaa, digitalisaation hallittua hyödyntämistä sekä yhteentoimivuutta.
4
5 Lajitietokeskuksen kokonaisarkkitehtuuri on kuvattu täällä: [[https:~~/~~/laji.fi/about/1357>>url:https://laji.fi/about/1357||shape="rect"]]
6
7 = System architecture =
8
9 Architecture from point of view of services and systems.
10
11 The image bellow shows some (but not all) FinBIF services and some of their relationships with each other and the user.
12
13 [[image:attach:architecture2021-05-26.png]]
14
15 Key services, systems and components
16
17 (% class="relative-table wrapped" style="width:99.9384%" %)
18 |=(((
19 Name
20 )))|=(((
21 Type
22 )))|=(% colspan="1" %)(% colspan="1" %)
23 (((
24 Techniques
25 )))|=(((
26 Description
27 )))|=(((
28 State
29 )))|=(((
30 Repository
31 )))|=(((
32 Documentation
33 )))
34 |(((
35 [[API.LAJI.FI>>url:https://api.laji.f||shape="rect"]]
36 )))|(((
37 HTTP-API,
38
39 Web-UI
40 )))|(% colspan="1" %)(% colspan="1" %)
41 (((
42 Loopback node.js framework
43
44 Swagger / OpenAPI specification
45
46 Swagger UI
47 )))|(((
48 api.laji.fi is the "frontend" of other APIs that provide most of it's services behind the scenes. The API also impelements some functionality by itself.
49 )))|(((
50 Production, version v0 beta
51 )))|(((
52 [[https:~~/~~/bitbucket.org/luomus/lajiapi>>url:https://bitbucket.org/luomus/lajiapi||shape="rect"]]
53 )))|(((
54 [[https:~~/~~/api.laji.fi/explorer/>>url:https://api.laji.fi/explorer/||shape="rect"]]
55 )))
56 |(% colspan="1" %)(% colspan="1" %)
57 (((
58 Laji.fi
59 )))|(% colspan="1" %)(% colspan="1" %)
60 (((
61 Web-UI
62 )))|(% colspan="1" %)(% colspan="1" %)
63 (((
64 Angular2 JS framework
65 )))|(% colspan="1" %)(% colspan="1" %)
66 (((
67 Frontend for Laji.fi and its sister sites.
68 )))|(% colspan="1" %)(% colspan="1" %)
69 (((
70 Production
71 )))|(% colspan="1" %)(% colspan="1" %)
72 (((
73 [[https:~~/~~/bitbucket.org/luomus/laji.fi-front>>url:https://bitbucket.org/luomus/laji.fi-front||shape="rect"]]
74 )))|(% colspan="1" %)(% colspan="1" %)
75 (((
76
77 )))
78 |(% colspan="1" %)(% colspan="1" %)
79 (((
80 Laji-Form.js
81 )))|(% colspan="1" %)(% colspan="1" %)
82 (((
83 Component
84 )))|(% colspan="1" %)(% colspan="1" %)
85 (((
86 React JS framework
87 )))|(% colspan="1" %)(% colspan="1" %)
88 (((
89 Laji-form is responsible for creating forms of Vihko Notebook.
90 )))|(% colspan="1" %)(% colspan="1" %)
91 (((
92 Production
93 )))|(% colspan="1" %)(% colspan="1" %)
94 (((
95 [[https:~~/~~/bitbucket.org/luomus/laji-form.js>>url:https://bitbucket.org/luomus/laji-form.js/src||shape="rect"]]
96
97 [[https:~~/~~/bitbucket.org/luomus/forms>>url:https://bitbucket.org/luomus/forms||shape="rect"]]
98 )))|(% colspan="1" %)(% colspan="1" %)
99 (((
100
101 )))
102 |(% colspan="1" %)(% colspan="1" %)
103 (((
104 Laji-Form
105 )))|(% colspan="1" %)(% colspan="1" %)
106 (((
107 Component
108 )))|(% colspan="1" %)(% colspan="1" %)
109 (((
110 PHP
111 )))|(% colspan="1" %)(% colspan="1" %)
112 (((
113 ??
114 )))|(% colspan="1" %)(% colspan="1" %)
115 (((
116 Production
117 )))|(% colspan="1" %)(% colspan="1" %)
118 (((
119 [[https:~~/~~/bitbucket.org/luomus/laji-form>>url:https://bitbucket.org/luomus/laji-form||shape="rect"]]
120 )))|(% colspan="1" %)(% colspan="1" %)
121 (((
122 [[https:~~/~~/bitbucket.org/luomus/laji-form>>url:https://bitbucket.org/luomus/laji-form||shape="rect"]]
123 )))
124 |(% colspan="1" %)(% colspan="1" %)
125 (((
126 Laji-Map
127 )))|(% colspan="1" %)(% colspan="1" %)
128 (((
129 Compontent
130 )))|(% colspan="1" %)(% colspan="1" %)
131 (((
132 Javascript
133 )))|(% colspan="1" %)(% colspan="1" %)
134 (((
135 Laji-Map is build on top of Leaflet map library and provides map services used by laji.fi, Vihko Notebook, etc.
136 )))|(% colspan="1" %)(% colspan="1" %)
137 (((
138 Production
139 )))|(% colspan="1" %)(% colspan="1" %)
140 (((
141 [[https:~~/~~/bitbucket.org/luomus/laji-map.js>>url:https://bitbucket.org/luomus/laji-map.js||shape="rect"]]
142 )))|(% colspan="1" %)(% colspan="1" %)
143 (((
144
145 )))
146 |(% colspan="1" %)(% colspan="1" %)
147 (((
148 LajiStore
149 )))|(% colspan="1" %)(% colspan="1" %)
150 (((
151 HTTP-API
152 )))|(% colspan="1" %)(% colspan="1" %)
153 (((
154 PHP
155 )))|(% colspan="1" %)(% colspan="1" %)
156 (((
157 Provides JSON storage for example for Vihko Notebook (used via api.laji.fi)
158 )))|(% colspan="1" %)(% colspan="1" %)
159 (((
160 Production
161 )))|(% colspan="1" %)(% colspan="1" %)
162 (((
163 [[https:~~/~~/bitbucket.org/luomus/lajistore-api>>url:https://bitbucket.org/luomus/lajistore-api||shape="rect"]]
164 )))|(% colspan="1" %)(% colspan="1" %)
165 (((
166
167 )))
168 |(% colspan="1" %)(% colspan="1" %)
169 (((
170 Laji-DW
171 )))|(% colspan="1" %)(% colspan="1" %)
172 (((
173 HTTP-API
174
175 Web-UI
176 )))|(% colspan="1" %)(% colspan="1" %)
177 (((
178 Java, Freemarker, jQuery
179 )))|(% colspan="1" %)(% colspan="1" %)
180 (((
181 Data Warehouse system provides ETL processes for observation, taxonomy, quality annotation data. It provides API for observation data primary sources to push data to the warehouse and "Pull readers" that read observation data from primary sources. It also implements several customized methods to fetch observation data from primary sources. Laji-DW provides query API that is used through [[api.laji.fi>>url:http://api.laji.fi||shape="rect"]] /warehouse endpoint. Finally it provides a management console for admins and citation viewer for performed observation data file downloads.
182 )))|(% colspan="1" %)(% colspan="1" %)
183 (((
184 Production
185 )))|(% colspan="1" %)(% colspan="1" %)
186 (((
187 [[https:~~/~~/bitbucket.org/luomus/laji-etl>>url:https://bitbucket.org/luomus/laji-etl/src||shape="rect"]]
188 )))|(% colspan="1" %)(% colspan="1" %)
189 (((
190
191 )))
192 |(% colspan="1" %)(% colspan="1" %)
193 (((
194 Laji-Auth
195 )))|(% colspan="1" %)(% colspan="1" %)
196 (((
197 HTTP-API
198
199 Web-UI
200 )))|(% colspan="1" %)(% colspan="1" %)
201 (((
202 Java Jersey
203 )))|(% colspan="1" %)(% colspan="1" %)
204 (((
205 [[login.laji.fi>>url:https://login.laji.fi||shape="rect"]] provides authentication methods for FinBIF services.
206 )))|(% colspan="1" %)(% colspan="1" %)
207 (((
208 Production
209 )))|(% colspan="1" %)(% colspan="1" %)
210 (((
211 [[https:~~/~~/bitbucket.org/luomus/laji-auth>>url:https://bitbucket.org/luomus/laji-auth||shape="rect"]]
212 )))|(% colspan="1" %)(% colspan="1" %)
213 (((
214
215 )))
216 |(((
217 Laji-Backend
218 )))|(((
219 HTTP-API
220 )))|(% colspan="1" %)(% colspan="1" %)
221 (((
222 Java
223 )))|(((
224 Internal API that provides some functionalities of api.laji.fi, for example /news, /information, taxon search and api related features.
225 )))|(((
226 Production, internal use only
227
228 (Will be partially phased out)
229 )))|(((
230 [[https:~~/~~/bitbucket.org/luomus/laji.fi-backend>>url:https://bitbucket.org/luomus/laji.fi-backend||shape="rect"]]
231 )))|(((
232
233 )))
234 |(((
235 Triplestore-API
236 )))|(((
237 HTTP-API
238 )))|(% colspan="1" %)(% colspan="1" %)
239 (((
240 Java
241 )))|(((
242 Rest HTTP API for managing an querying contents of Triplestore Ontology database, that contains information for example about taxonomy, persons, image metadata, etc.
243 )))|(((
244 Production, internal use only
245 )))|(((
246 [[https:~~/~~/bitbucket.org/luomus/triplestore>>url:https://bitbucket.org/luomus/triplestore||shape="rect"]]
247 )))|(((
248
249 )))
250 |(((
251 Triplestore Editor
252 )))|(((
253 Web-UI
254 )))|(% colspan="1" %)(% colspan="1" %)
255 (((
256 Java, Freemarker, jQuery
257 )))|(((
258 Web UI for managing contents of Triplestore Ontology database for IT-admins.
259 )))|(((
260 Production, admin tool
261 )))|(((
262 [[https:~~/~~/bitbucket.org/luomus/triplestore>>url:https://bitbucket.org/luomus/triplestore||shape="rect"]]
263 )))|(((
264
265 )))
266 |(((
267 Taxon Editor
268 )))|(((
269 Web-UI
270 )))|(% colspan="1" %)(% colspan="1" %)
271 (((
272 Java, Freemarker, jQuery
273 )))|(((
274 Web UI for managing taxonomy content of Triplestore Ontology database.
275 )))|(((
276 Production
277 )))|(((
278 [[https:~~/~~/bitbucket.org/luomus/triplestore>>url:https://bitbucket.org/luomus/triplestore||shape="rect"]]
279 )))|(((
280
281 )))
282 |(((
283 tun.fi / id redirect servce
284 )))|(((
285 HTTP-API
286 )))|(% colspan="1" %)(% colspan="1" %)
287 (((
288 Java
289 )))|(((
290 Provides machine readable and human readable redirect service for HTTP URI Identifiers used by FinBIF for all things such as taxa, observations, persons, areas, collections. [[https:~~/~~/www.w3.org/TR/ld-bp/#HTTP-URIS>>url:https://www.w3.org/TR/ld-bp/#HTTP-URIS||shape="rect"]]
291 )))|(((
292 Production
293 )))|(((
294 [[https:~~/~~/bitbucket.org/luomus/id>>url:https://bitbucket.org/luomus/id||shape="rect"]]
295 )))|(((
296
297 )))
298 |(((
299 Kotka
300 )))|(((
301 Web-UI
302 )))|(% colspan="1" %)(% colspan="1" %)
303 (((
304 PHP
305 )))|(((
306 Kotka collection management system
307 )))|(((
308 Production
309
310 (will be reimplemented using laji.fi/notebook technologies)
311 )))|(((
312 [[https:~~/~~/bitbucket.org/luomus/kotka>>url:https://bitbucket.org/luomus/kotka||shape="rect"]]
313 )))|(((
314
315 )))
316 |(((
317 Pyha
318 )))|(((
319 WEB-UI
320 )))|(% colspan="1" %)(% colspan="1" %)
321 (((
322 Python Django
323 )))|(((
324 Data request service that is used to request and provide access to closed/not-coarsed observation data.
325 )))|(((
326 Production
327 )))|(((
328 [[https:~~/~~/bitbucket.org/luomus/pyha>>url:https://bitbucket.org/luomus/pyha||shape="rect"]]
329 )))|(((
330
331 )))
332 |(((
333 Multimedia-API
334
335
336 )))|(((
337 HTTP-API
338 )))|(% colspan="1" %)(% colspan="1" %)
339 (((
340 Java, Imagemagick
341 )))|(((
342 Provides HTTP API to post images (in future audio, videos?) and their metadata and to query images. Converts thumbnails etc. Used by api.laji.fi.
343 )))|(((
344 Production
345 )))|(((
346 [[https:~~/~~/bitbucket.org/luomus/kuvapalvelu-server>>url:https://bitbucket.org/luomus/kuvapalvelu-server||shape="rect"]]
347 )))|(((
348
349 )))
350 |(((
351 Image-Storage
352 )))|(((
353 Web server
354 )))|(% colspan="1" %)(% colspan="1" %)
355 (((
356 Nginx (hosting), Python (API)
357 )))|(((
358 Provides physical image storage
359 )))|(((
360 Production
361 )))|(((
362 internal only
363 )))|(((
364
365 )))
366 |(((
367 GIS
368 )))|(((
369 WFS / WMTS
370 )))|(% colspan="1" %)(% colspan="1" %)
371 (((
372 Geoserver
373 )))|(((
374 Provides several layers and in future observation data in GIS formats
375 )))|(((
376 Production
377 )))|(((
378 internal only
379 )))|(((
380
381 )))
382 |(((
383 GIS converter
384 )))|(((
385 HTTP-API
386 )))|(% colspan="1" %)(% colspan="1" %)
387 (((
388 R
389 )))|(((
390 Transforms downloads to GIS formats
391 )))|(((
392 Production
393 )))|(((
394 [[https:~~/~~/github.com/luomus/finbif-geo-convert>>url:https://github.com/luomus/finbif-geo-convert||shape="rect"]]
395 )))|(((
396
397 )))
398 |(((
399 FinBIF 2 GBIF
400 )))|(((
401 HTTP-API
402 )))|(% colspan="1" %)(% colspan="1" %)
403 (((
404 R
405 )))|(((
406 Shares FinBIF data to [[https:~~/~~/www.gbif.org/>>url:https://www.gbif.org/||shape="rect"]]
407 )))|(((
408 Production
409 )))|(((
410 [[https:~~/~~/github.com/luomus/finbif2gbif>>url:https://github.com/luomus/finbif2gbif||shape="rect"]]
411 )))|(((
412
413 )))
414 |(((
415
416 )))|(((
417
418 )))|(% colspan="1" %)(% colspan="1" %)
419 (((
420
421 )))|(((
422
423 )))|(((
424
425 )))|(((
426
427 )))|(((
428
429 )))
430 |(% colspan="1" %)(% colspan="1" %)
431 (((
432
433 )))|(% colspan="1" %)(% colspan="1" %)
434 (((
435
436 )))|(% colspan="1" %)(% colspan="1" %)
437 (((
438
439 )))|(% colspan="1" %)(% colspan="1" %)
440 (((
441
442 )))|(% colspan="1" %)(% colspan="1" %)
443 (((
444
445 )))|(% colspan="1" %)(% colspan="1" %)
446 (((
447
448 )))|(% colspan="1" %)(% colspan="1" %)
449 (((
450
451 )))
452 |(% colspan="1" %)(% colspan="1" %)
453 (((
454
455 )))|(% colspan="1" %)(% colspan="1" %)
456 (((
457
458 )))|(% colspan="1" %)(% colspan="1" %)
459 (((
460
461 )))|(% colspan="1" %)(% colspan="1" %)
462 (((
463
464 )))|(% colspan="1" %)(% colspan="1" %)
465 (((
466
467 )))|(% colspan="1" %)(% colspan="1" %)
468 (((
469
470 )))|(% colspan="1" %)(% colspan="1" %)
471 (((
472
473 )))
474
475
476 = Data storage architecture =
477
478 Architecture from point of view of data strorage.
479
480 (% class="relative-table wrapped" style="width:47.6279%" %)
481 |=(((
482 System
483 )))|=(((
484 Data
485 )))|=(((
486 Technique
487 )))|=(((
488 Data type
489 )))
490 |(((
491 Kotka
492 )))|(((
493 Specimen primary data storage
494 )))|(((
495 Oracle
496 )))|(((
497 Triplets/Ontology
498 )))
499 |(% colspan="1" %)(% colspan="1" %)
500 (((
501 Kotka
502 )))|(% colspan="1" %)(% colspan="1" %)
503 (((
504 Kotka internal search
505 )))|(% colspan="1" %)(% colspan="1" %)
506 (((
507 ElasticSearch
508 )))|(% colspan="1" %)(% colspan="1" %)
509 (((
510 JSON
511 )))
512 |(((
513 Vihko
514 )))|(((
515 Observation primary data storage
516 )))|(((
517 Oracle
518 )))|(((
519 JSON
520 )))
521 |(((
522 Master data
523 )))|(((
524 Taxonomy, Area, Collection metada etc storage location
525 )))|(((
526 Oracle
527 )))|(((
528 Triplets/Ontology
529 )))
530 |(((
531 Multimedia-API
532 )))|(((
533 Image (+3d, audio) storage
534 )))|(((
535 Disk
536 )))|(((
537 Files
538 )))
539 |(((
540 Laji-DW
541 )))|(((
542 ETL process and metadata
543 )))|(((
544 Oracle
545 )))|(((
546 Relational database
547 )))
548 |(% colspan="1" %)(% colspan="1" %)
549 (((
550 Laji-DW
551 )))|(% colspan="1" %)(% colspan="1" %)
552 (((
553 Observation Search Engine
554 )))|(% colspan="1" %)(% colspan="1" %)
555 (((
556 Vertica
557 )))|(% colspan="1" %)(% colspan="1" %)
558 (((
559 Column storage relational database
560 )))
561 |(% colspan="1" %)(% colspan="1" %)
562 (((
563 Taxa
564 )))|(% colspan="1" %)(% colspan="1" %)
565 (((
566 Taxonomy endpoint
567 )))|(% colspan="1" %)(% colspan="1" %)
568 (((
569 ElasticSearch
570 )))|(% colspan="1" %)(% colspan="1" %)
571 (((
572 JSON
573 )))
574 |(% colspan="1" %)(% colspan="1" %)
575 (((
576 Geo
577 )))|(% colspan="1" %)(% colspan="1" %)
578 (((
579 Georeferencing and geo queries
580 )))|(% colspan="1" %)(% colspan="1" %)
581 (((
582 ElasticSearch
583 )))|(% colspan="1" %)(% colspan="1" %)
584 (((
585 JSON
586 )))
587 |(% colspan="1" %)(% colspan="1" %)
588 (((
589 GIS
590 )))|(% colspan="1" %)(% colspan="1" %)
591 (((
592 Geoserver data
593 )))|(% colspan="1" %)(% colspan="1" %)
594 (((
595 PostgreSQL
596 )))|(% colspan="1" %)(% colspan="1" %)
597 (((
598 Relational database
599 )))
600
601 = Long-term archiving =
602
603 FinBIF preservation strategy involves maintaining biodiversity occurrence data in the FinBIF data format, aligned with the international ABCD (Access to Biological Collections Data) standard. Metadata adheres to the EML (Ecological Metadata Language) standard, promoting compatibility and standardized data curation. In instances of format changes, FinBIF oversees necessary data migrations.
604
605 Primary data responsibility resides with FinBIF, while data owners retain responsibility for secondary copies. Data owners, however, retain control over their data, including the right to request removal. In such cases, citable downloads are preserved indefinitely in text format to maintain persistent citation. Identifiers, once assigned, are kept persistent and are never reassigned.
606
607 Data is currently hosted and backed up by the University of Helsinki IT services. We are in the process of extending collaboration with CSC's (Finnish IT Center for Science) Finnish National Digital Preservation Service.
608
609 = Server architecture =
610
611 FinBIF services are hosted by Helsinki University IT Center and CSC Pouta cloud and OpenShift Kubernetes services.
612
613
614