Flow of meteorological, flux and soil data into databases

Introduction

Data from SMEAR II is processed automatically for

• Online graphs for monitoring of data quality and for the SMEAR webpage
• SMEAR database
• ICOS Ecosystem and Atmospheric Thematic Centres, data eventually available at Carbon Portal

The online graphs are showing either 30 min averages or momentary data. Soil variables (temperature, moisture, water potential, heat flux) are horizontally (layer-wise) averaged vertical profiles.

Time resolution of the data in SMEAR database is one minute except turbulent fluxes are 30 min averages. Soil variables are layer-wise averaged vertical profiles. Below-canopy PAR is average of each sensor array (maapar, 3xcanpar).

The Pre-ICOS ecosystem data consisted of 30 min averages and instantaneous records (1 min averages in case of very frequent measurements) of sensor-wise data, i.e. soil variables and below-canopy PAR measured at different locations are submitted as separate variables. The data flow is continuation of past projects such as IMECC, InGOS, GHG-Europe and Carboextreme.

ICOS ecosystem data is relatively unprocessed sensor-wise meteo, radiation and soil data.

ICOS atmospheric data consists of CO2, H2O, CH4 and CO concentration profiles (Picarro G2401) and 1-min meteorological data (T, RH, WS, WD) taken at 16.8 m, 67.2 m and 125 m.

SMEAR databases and online graphs

Near real time data

All the relevant data is copied hourly to disk servers in Helsinki by script hippuko on server timo in Hyytiälä.

Data processing server hippuko.atm.helsinki.fi runs set of data processing scripts hourly or daily. For instance, one minute and half hour averages are calculated from mast, radiation tower, present weather sensor and below-canopy radiation measurements as well as soil temperature and moisture.

The online processed data are saved in intermediate files on hippuko and inserted to MySQL database 'smear' at db5.csc.fi, tables 'HYY_META', 'HYY_EDDY233','HYY_EDDYTOW', 'HYY_EDDYSUB' and 'HYY_DMPS'...

Quality-checked data

Online-processed data in the databases are replaced every couple of months with data having up-to-date calibration corrections, visual quality check etc.
 The processed meteorological and soil data are inserted manually into MySQL databases at db5.csc.fi.
Some of the data update scripts can be found at \\aella.ad.helsinki.fi\h528\smear\hyytiala\matlab\database.

Pre-ICOS (RETIRED)

Near real time data

Selection of meteorological, soil and flux variables is sent daily to ICOS ETC (European Fluxes Database). See description of ICOS ATC greenhouse gas data here.

Scheduled task is launched on Hippuko every morning.
 It collects

• half-hourly averaged data (ave=30)
• data at original (1 min) time resolution (ave=1)

from before mentioned intermediate data compiled by the online processing scripts. The soil and below-canopy PAR data are sensor-wise (for SMEAR database the scripts produce spatially averaged data) instantaneous records and 30 min averages. Final product is daily datafiles in comma-separated text format. The result files also include two header rows with variable names and units. These are defined in files icos_columns.txt.
 Unlike data in atmo_monthfiles, SMEAR database etc. the date&time columns in ICOS NRT data indicate the end of the averaging period!

Raw eddy233 data (24 files) from previous day are copied from atm-raw into daily zip archive on Hippuko.

Phenology camera picture taken at 12 on previous day is also copied to Hippuko.

The data and phenology camera pictures are then synced to European Fluxes database (ftp addresses //gaia.agraria.unitus.it/imeccreal/fi-hyy and //gaia.agraria.unitus.it/icosnrt/Fi-Hyy).

Variables in ICOS ETC NRT data are listed below. Latest version of the instrument metadata file can be found at
\\aella.ad.helsinki.fi\h528\smear\hyytiala\Database_EU\GHG-Europe_InGOS_ICOS\Fi_Hyy_BADM_YYYYMMDD.xls
 This is the metadata format used for all data submitted to European Fluxes Database.

Quality-checked data

Consolidated dataset of previous year is submitted to ICOS ETC once a year (early spring).