Flow of meteorological, flux and soil data into databases

Introduction

Data from SMEAR II is processed automatically for

• Online graphs for monitoring of data quality and for the SMEAR webpage
• SMEAR database
• ICOS Ecosystem and Atmospheric Thematic Centres, data eventually available at Carbon Portal
• Other Finnish and international research infrastructure and project databases

The online graphs are showing either 30 min averages or momentary data from each instrument.

Time resolution of the data in SMEAR database is one minute except turbulent fluxes are 30 min averages. Soil variables are layer-wise averaged vertical profiles. Below-canopy PAR is average of each sensor array (maapar, 3 x canpar, 4 x ICOS plots).

The Pre-ICOS ecosystem data consisted of 30 min averages and instantaneous records (1 min averages in case of very frequent measurements) of sensor-wise data, i.e. soil variables and below-canopy PAR measured at different locations are submitted as separate variables. The data flow is continuation of past projects such as IMECC, InGOS, GHG-Europe and Carboextreme.

ICOS ecosystem data is relatively unprocessed sensor-wise meteo, radiation and soil data.

ICOS atmospheric data consists of CO2, H2O, CH4 and CO concentration profiles (Picarro G2401) and 1-min meteorological data (T, RH, WS, WD) taken at 16.8 m, 67.2 m and 125 m.

SMEAR databases and online graphs

Near real time data

All the relevant data are copied several times per day to disk servers in Helsinki by scripts on data processing server hippuko-22.atm.helsinki.fi in Helsinki.

Hippuko runs set of data processing scripts at different intervals from hourly to daily. For instance, one minute and half-hourly averages are calculated from mast, radiation tower, present weather sensor and below-canopy radiation measurements as well as soil temperature and moisture.

The online processed data are saved in intermediate files on hippuko and inserted to MySQL database 'smear' at db5.csc.fi, tables 'HYY_META', 'HYY_EDDY233','HYY_EDDYTOW', 'HYY_EDDYSUB', 'HYY_AERO' and 'HYY_DMPS'.

Quality-checked data

Online-processed data in the databases are replaced every couple of months or at least once per year with data having up-to-date calibration corrections, visual quality check, and removal of known periods of instrument malfunction etc. The processed data are inserted manually into SMEAR database.

Some of the data processing and update scripts can be found at smear group disk folder hyytiala/matlab. The codes might not be up to date though, better ask recent versions from atm-data@ if you don't have access to hippuko.

ICOS

Near real time data

Collection of data is run on timo instead of hippuko to make the process more robust. Matlab and other scripts in /home/smear/matlab and subfolders take care of compiling the data and producing diagnostic graphics couple of times per day. Most recent data are sent to Carbon Portal via http API or to ATC via SFTP every morning. See more detailed documentation here and on timo.

SMEAR II staff updates ICOS metadata at ETC and ATC whenever instruments or their configuration are changed.

Quality-checked data

For Class 1 and 2 stations (SMEAR II, Siikaneva 1), consolidated level 2 datasets from the beginning of ICOS (2017 for atmospheric data, 2018 for ecosystem data) are compiled twice per year by ETC and ATC and distributed via Carbon Portal.

Data and metadata for associated ecosystem stations (Kuivajärvi) are compiled yearly by INAR and submitted manually to ETC.

Pre-ICOS (RETIRED)

Near real time data

Selection of meteorological, soil and flux variables is sent daily to ICOS ETC (European Fluxes Database). See description of ICOS ATC greenhouse gas data here.

Scheduled task is launched on Hippuko every morning.
 It collects

• half-hourly averaged data (ave=30)
• data at original (1 min) time resolution (ave=1)

from before mentioned intermediate data compiled by the online processing scripts. The soil and below-canopy PAR data are sensor-wise (for SMEAR database the scripts produce spatially averaged data) instantaneous records and 30 min averages. Final product is daily datafiles in comma-separated text format. The result files also include two header rows with variable names and units. These are defined in files icos_columns.txt.
 Unlike data in atmo_monthfiles, SMEAR database etc. the date&time columns in ICOS NRT data indicate the end of the averaging period!

Raw eddy233 data (24 files) from previous day are copied from atm-raw into daily zip archive on Hippuko.

Phenology camera picture taken at 12 on previous day is also copied to Hippuko.

The data and phenology camera pictures are then synced to European Fluxes database (ftp addresses //gaia.agraria.unitus.it/imeccreal/fi-hyy and //gaia.agraria.unitus.it/icosnrt/Fi-Hyy).

Variables in ICOS ETC NRT data are listed below. Latest version of the instrument metadata file can be found at
\\aella.ad.helsinki.fi\h528\smear\hyytiala\Database_EU\GHG-Europe_InGOS_ICOS\Fi_Hyy_BADM_YYYYMMDD.xls
 This is the metadata format used for all data submitted to European Fluxes Database.

Quality-checked data

Consolidated dataset of previous year is submitted to ICOS ETC once a year (early spring).