Data flow

Last modified by ppaalto@helsinki_fi on 2024/02/07 06:33

File servers

  • vakka
    • At the moment /wrk-vakka/group/atm/backup/ is the same as atm-raw. Data is copied here from atm-raw. In future this will replace atm-raw
    • /wrk-vakka/group/atm/ARCHIVE/ has some old data
  • //nas-fs.cs.helsinki.fi/phys-atm/atm-raw
    • Primary place for large datasets
  • //group5.ad.helsinki.fi/h527/atm_campaigns
    • For projects producing small amounts of data
  • //group5.ad.helsinki.fi/h527/smear
    • The original place for SMEAR data. Is still used for small data sets
  • datacloud
    • Serves as place for project data. Data is also copied here from outside UHEL network and also shared with some people outside UHEL
    • Has place also for old arhived data, same as vakka
  • CSC IDA and database
    • IDA hosts some common datasets updated by yearly basis
    • CSC database is the main source of processed data https://smear.avaa.csc.fi/ is using this database

Computers

  • hippuko.atm
    • Main computer for data analysis and data flow
    • Copies data (rsync) from eddy2.atm, Hyytiälä, Värriö and lab-atm-smear vlan computers to the file servers mentioned above
    • Processing data and visualizing it mainly with Matlab
    • Copies data pictures to www.atm
    • Copies data to CSC database
    • Copies data to infrastructure (ICOS, ACTRIS) servers outside UHEL
    • timed jobs run under ymp_fyl account
  • eddy2.atm (physical linux server Centos7, 130G memory, 40T disk space, 32 cores )
    • Data outside UHEL network is copied here with sftp. Has now around 10T of data in temporary storage between the measurement computer and final storage (like atm-raw)
    • Some data analysis work
    • Some small scientific compting  tasks
    • ssh port open to world
    • trajectory calculations, hosting meteorological data (~10T) for that
  • www.atm
    • Hosting data pictures and some small amounts of data shared outside UHEL
  • grafana.atm.helsinki.fi
    • New server to visualize data with Grafana and InfluxDB
  • lablog.atm
    • Server holding  elog server for measurement diaries
  • Hyytiälä server (physical linux server)
    • Collects data from Hyytiälä measurement computers, has ssh server for data transfer
    • Is doing some data analysis work and copying data to atm-raw, smear amd Datacloud. Copies data also to ICOS infrastructure server. Also copies data to CSC database
    • timed jobs run under smear (local account) username
  • Värriö server (physical windows server)
    • Collects data from Värriö measurement computers, has ssh server for data transfer
    • timed jobs run under smear (local account) account

Locations

  • lab-atm-smear vlan
    • This is a vlan, which has measurement computers in Kumpula and modem-vpn connected measurement computers elsewhere
  • Hyytiälä (SMEAR)
    • Hyytiälä has it's own laboratory network with many measurement computers
  • Värriö (SMEAR)
    • Värriö has it's own laboratory network with around ten measurement computers
  • Outside UHEL
    • These are permanent or campaign sites outside UHEL network