Skip to Content

Data flow

Last modified by Pasi Aalto on 2025/11/10 08:12

File servers

  • vakka
    • Primary place for large datasets
    • /wrk-vakka/group/atm/backup/ 
    • /wrk-vakka/group/atm/ARCHIVE/ has some old data
  • //group5.ad.helsinki.fi/h527/atm_campaigns
    • For projects producing small amounts of data
  • //group5.ad.helsinki.fi/h527/smear
    • The original place for SMEAR data. Is still used for small data sets
  • datacloud
    • Serves as place for project data. Data is also copied here from outside UHEL network and also shared with some people outside UHEL
    • Has place also for old arhived data, same as vakka
  • CSC IDA and database
    • IDA hosts some common datasets updated by yearly basis
    • CSC database is the main source of processed data https://smear.avaa.csc.fi/ is using this database

Computers

  • hippuko.atm
    • Main computer for data analysis and data flow
    • Copies data (rsync) from eddy2.atm, Hyytiälä, Värriö and lab-atm-smear vlan computers to the file servers mentioned above
    • Processing data and visualizing it mainly with Matlab
    • Copies data pictures to www.atm
    • Copies data to CSC database
    • Copies data to infrastructure (ICOS, ACTRIS) servers outside UHEL
    • timed jobs run under copier account
  • icos.atm
    • Same as hippuko.atm, but dedicated to ICOS tasks
  • eddy2.atm (physical linux server Centos7, 130G memory, 40T disk space, 32 cores )
    • Data outside UHEL network is copied here with sftp. Has now around 10T of data in temporary storage between the measurement computer and final storage (like atm-raw)
    • Some data analysis work
    • Some small scientific compting  tasks
    • ssh port open to world
    • trajectory calculations, hosting meteorological data (~10T) for that
  • www.atm
    • Hosting data pictures and some small amounts of data shared outside UHEL
  • grafana.atm.helsinki.fi
    • Server to visualize data with Grafana and InfluxDB
  • lablog.atm
    • Server holding  elog server for measurement diaries
  • sftp.atm
    • This server is used to move data outside UHEL network
  • mqtt.atm and mqtt-client.atm
    • MQTT data flow to vakka and other servers
  • Hyytiälä server mycos.local.lab (physical linux server)
    • Collects data from Hyytiälä measurement computers, has ssh server for data transfer
    • Is doing some data analysis work and copying data to vakka, smear and Datacloud. Copies data also to ICOS infrastructure server. Also copies data to CSC database
    • timed jobs run under smear (local account) username

Locations

  • lab-atm-smear vlan
    • This is a vlan, which has measurement computers in Kumpula and modem-vpn connected measurement computers elsewhere
  • Hyytiälä (SMEAR)
    • Hyytiälä has it's own laboratory network (old and smear-hyytiala vlan) with many measurement computers. 
  • Värriö (SMEAR)
    • Värriö has it's own laboratory network lab-atm-varrio with around ten measurement computers
  • Outside UHEL
    • These are permanent or campaign sites outside UHEL network