Skip to Content

Flow of meteorological, flux and soil data into databases

Last modified by Pasi Kolari on 2026/10/20 19:29

Introduction

Data from SMEAR II are processed automatically for

  • Online graphs for monitoring of data quality and for the SMEAR webpage
  • SMEAR database
  • ICOS Ecosystem and Atmospheric Thematic Centres, data eventually available at Carbon Portal
  • Other Finnish and international research infrastructure and project databases

The online graphs are showing either 30 min averages or momentary data from each instrument.

Time resolution of the data in SMEAR database is one minute except turbulent fluxes are 30 min averages. Soil variables are layer-wise averaged vertical profiles. Below-canopy PAR is average of each sensor array (maapar, 3 x canpar, 4 x ICOS plots).

The pre-ICOS ecosystem data consisted of 30 min averages and instantaneous records (1 min averages in case of very frequent measurements) of sensor-wise data, i.e. soil variables and below-canopy PAR measured at different locations are submitted as separate variables. The data flow is continuation of past projects such as IMECC, InGOS, GHG-Europe and Carboextreme.

ICOS ecosystem data are relatively unprocessed sensor-wise meteo, radiation and soil data.

ICOS atmospheric data consist of CO2, H2O, CH4 and CO concentration profiles (Picarro G2401) and 1-min meteorological data (T, RH, WS, WD) taken at 16.8 m, 67.2 m and 125 m.

SMEAR databases and online graphs

Near real time data

All the relevant data are copied several times per day to disk servers in Helsinki by scripts on data processing server hippuko-22.atm.helsinki.fi in Helsinki.

Hippuko runs set of data processing scripts at different intervals from hourly to daily. For instance, one minute and half-hourly averages are calculated from mast, radiation tower, present weather sensor and below-canopy radiation measurements as well as soil temperature and moisture.

The online processed data are saved in intermediate files on hippuko and inserted to MySQL database 'smear' at db5.csc.fi, tables 'HYY_GAS', 'HYY_META', 'HYY_RAD', 'HYY_SOIL', 'HYY_EDDYMAST','HYY_EDDYTOW', 'HYY_EDDYSUB', 'HYY_AERO' and 'HYY_DMPS'.

Quality-checked data

Online-processed data in the databases are replaced at least once per year with data having up-to-date calibration corrections, visual quality check, and removal of known periods of instrument malfunction etc. The processed data are inserted manually into SMEAR database.

Some of the data processing and update scripts can be found at smear group disk folder hyytiala/matlab. The codes might not be up to date though, better ask recent versions from atm-data@ if you don't have access to hippuko.

ICOS

Near real time data

Collection of data is run on icos-25 instead of hippuko to make the process more robust. Matlab and other scripts in /work/matlab and subfolders take care of compiling the data and producing diagnostic graphics couple of times per day. Most recent data are sent to Carbon Portal via http API or to ATC via SFTP every morning. See more detailed documentation here and on timo.

SMEAR II staff updates ICOS metadata at ETC and ATC whenever instruments or their configuration are changed.

Quality-checked data

For Class 1 and 2 stations (SMEAR II, Siikaneva 1), consolidated level 2 datasets from the beginning of ICOS (2017 for atmospheric data, 2018 for ecosystem data) are compiled twice per year by ETC and ATC and distributed via Carbon Portal.

Data and metadata for associated ecosystem stations (Kuivajärvi) are compiled yearly by INAR and submitted manually to ETC.