SMEAR II and Siikaneva 1 ICOS dataflow
SMEAR II is ICOS Class 1 and Siikaneva 1 Class 2 ecosystem station. Flux, meteo and soil data are sent daily (Near Real Time) to ICOS Carbon Portal.
The processing scripts are located on timo /home/smear/matlab/icos_es. Backup on hippuko /home/ymp_fyl/matlab/siikaneva1.
The folder contains, for instance, Matlab scripts/functions
icos_es.m: main script that calls the processing and plotting functions
prepare_icos_bm_data.m: function for reading data from raw meteo and soil datafiles, processing to physical units and converting to ICOS format
process_icos_ec_data.m: function for reading data from raw EC datafiles and converting to ICOS format
FI-Hyy_dataconfig.xlsx and FI-Sii_dataconfig.xlsx:
meteo data configuration parameters like source file paths and filenames, variable mapping to file columns, result file IDs etc.
See instructions for using the files on the INFO sheets of the files.
There are also some specific tool functions, plotting scripts and functions to insert some ICOS variables to SMEAR database (most SMEAR II ICOS measurements are processed for SMEAR database elsewhere, on hippuko).
FI-Hyy_cal*.csv and FI-Sii_cal*.csv contain the conversion parameters from raw electric units (if any) to physical units. See instructions for using the file inside the file itself.
The configuration and cal files must be updated promptly if there's changes in instrument configuration. See readme file on timo for typical cases where metadata update is needed.
The ICOS format files are sent to Carbon Portal using shell script icosetcftp6.
Type 'icosetcftp6 -h' in terminal for help.
The scripts are run automatically every morning by cron job that runs shell script runmatlab. To ensure that there are no gaps in the data due to data transfer from Siikaneva computers to timo, data are always processed couple of days backwards. By default, icosetcftp6 checks data from the last 6 days and resubmits changed files.
The processing scripts produce graphics for all ICOS variables, summary figures of meteo and soil variables and some additional diagnostic data. The plots are saved on the web server, some summary and diagnostic plots also on Hyytiälä Online data page.