Page tree
Skip to end of metadata
Go to start of metadata

Please update your links to the updated Ukko2 User Guide.

The new HPC cluster ukko2 is now ready for production. To be able to use the cluster, you need to either

  • be member of CS staff or
  • belong to the IDM group grp-cs-ukko2

Connections are only allowed from the domain, however using VPN or Eduroam, is not sufficient.
To access ukko2 outside of the domain (e.g. home), add following to your ~/.ssh/config

   ProxyCommand ssh -W %h:%p

(Note: with OpenSSH versions > 7.3, you can substitute the ProxyCommand with more human readable line: ProxyJump

The login node is only meant for batch job management, so you should not execute any jobs on it. To access the node:

ssh <username>

Batch scheduling system is the most prominent difference between ukko1 and ukko2. Instead of logging into computing node and executing jobs there interactively, you log in to the login node and submit the jobs via a batch scheduling system. You may submit as many jobs to queues as you like.

Another change is the module system. Software packages, compiler environments etc are handled by modules. This allows user to load, unload or change between environments flexibly.

Ukko 2 Nodes

  • Single login node ukko2 serves logins, compiler environments and all batch scheduling functions
  • 31 regular compute nodes (ukko-02  - ukko-32):   28 cores, 2 threads and  256 GB RAM. 
  • 2 big memory  nodes (ukko2-pekka, ukko2-paavo):  96 cores, 2 threads and 3 TB RAM.  
  • 2 GPU nodes (ukko2-g01, ukko2-g02) :  28 cores, 2 threads, 512 GB RAM andTesla P100 GPU cards. 

I/O, disks and filesystems

Ukko2 shares home directory with Ukko1: all your files will be available on the new cluster. You can also access the files in Ukko from the department computers using the path /cs/work/home/username.

Queuing system

Jobs are managed with the SLURM batch scheduling system, and the runtime environment is managed as modules that will allow for choosing the correct compilers, libraries and tools.

The jobs are started from the ukko2 login node (

There are five production queues and one short queue to test jobs, some queues overlap others for better system utilisation:

Queue nameWall Time limitCoresMemory per coreNotes
short24h10608Gb - 32Gb
long14 days8688Gb
extralong60 days288Gb
bigmem7 days5632Gb
gpu7 days19218GbHave to reserve with #SBATCH -p gpu
test1h1128GbHave to reserve with #SBATCH -p test

Testing and development

Before running jobs on the production queues, resource requirements and scalability should be tested. 1h test queue is available for this purpose.  When submitting a job for a test, following parameters should be set in the batch -file to find out the job performance. Mail parameters can of course be used also with non test jobs:

#SBATCH -p test             // define test partition
#SBATCH --mail-type=END     // END, or ALL are useful here 
#SBATCH -e errors.err       // stderror
#SBATCH -o output.file      // output file
#SBATCH -c 1                // number of CPU's (Cores) requested
echo "Hello SLURM"          // commands to be run

User may also specify mail address other than default (system default is the user submitting the job):

#SBATCH --mail-user <mail@address>

Interactive use

There is no need for direct access to node to perform interactive session. Slurm allows interactive sessions to be started by using srun. After entering the srun command, interactive job request is sent to the normal queue to wait for resources to become available. Once resources are available, the session starts on a compute node, and you are put into the directory from which you ran the launched the session. You can then run commands.

Environment you get on compute node is determined by a combination of three things:

  1. The environment as set in your session from which you launch the srun command.
  2. Any extra variables set by Slurm
  3. Settings from your .bashrc file

Example of starting 1 core, 1 task interactive session with bash -shell.

srun -c 1 --ntasks-per-node=1 --pty bash

At this time, there are no nodes reserved  for pure interactive use, and there may be some wait time - especially if resource request is significant.

Advance Reservations

For advance reservations, please contact the IT suppport.
Slurm has a full support for Advance Reservations which mean that user, or user group may ask for a dedicated time slot and resources for the slot and then execute jobs at the dedicated slot.

Batch job resources

Job execution priorities depend upon user resource requests. If no resource limits are requested in the batch script, then system and queue defaults are used. Job priority and scheduling decisions are based on available system resources. Most "expensive" resource being first on the list:

  1. GPU requested
  2. Memory requested
  3. Wall Time requested
  4. CPU's requested

The fewer resources a job requests,  the faster it goes into execution.

Please Note: System defaults: Job placement in short queue, allocation of 512MB of memory per CPU (core), 1 hour of Wall Time and 1 CPU (core). Defaults can be changed by user specified resource requests in the job batch script (See Batch files below).

For resource allocation purposes CPU=Core.

Finding job resource utilisation

Finding actual memory usage of application may be challenge at times. Good news are that Slurm features simple utility to provide job utilisation details from any job that has completed. Using this utility helps to determine actual resource needs of the job for future reference.

seff <jobID>

Please Note: Accurate resource requests (memory, CPU and time) will expedite the execution of your jobs - and lead to better system utilisation.

Finding actual resource usage from job accounting 

Slurm has powerful accounting feature with myriad options to choose from. Among many other things, it is possible to create output from the user jobs featuring some of the more useful details:

sacct -oJobID,JobName,ExitCode,NNodes,NCPUS,MaxRSS,Elapsed,End

Provides easy to read list formatted output, where fields are:

JobID: Job identification number

JobName: Job name given in the Slurm batch script

ExitCode: Exit code once job was terminated

NNodes: Node count

NCPUS: CPU's (Core) reserved by the job

MaxRSS: Memory peak usage during job execution, returns value when job has finished. This value can be used to adjust the requested memory value in the batch script accordingly.

Elapsed: Time batch job was in execution

End: End time of batch job

Other examples:

Lists details when JobID is known:

sacct -j <jobID> -oJobID,JobName,ExitCode,NNodes,NCPUS,MaxRSS,Elapsed,End

Jobs listed by UserID:

sacct -u <userID> -oJobID,JobName,ExitCode,NNodes,NCPUS,MaxRSS,Elapsed,End

Additional accounting options, and parameters can be found from the accounting man page

man sacct

Batch files

To run a job you will need to write a batch file.

A batch file begins with the shell to be used and otherwise consists of slurm options and linux commands to be executed. The slurm options may also be specified as command line arguments (or both; in that case the command line options take precedence).

Please Note: Unless you need GPU, or test queue there is no need to specify the queue with -p parameter. In case gpu resources are desired, then following line needs to be added to the Slurm batch file:

#SBATCH -p gpu

If test queue is desired, then:

#SBATCH -p test

Under normal circumstances Slurm will handle the production job placement to appropriate queues based on the resource requests in the batch file. 

Setting up batch job e-mail notifications

It is possible to set e-mail notifications for batch job. If set, user is notified about changes in the job status.

Please Note: Job END and ALL -notifications includes the seff output once batch job has finished, and is very useful for finding out the actual job resource utilisation. 

Most commonly chosen mail options are: NONE, BEGIN, END, FAIL or ALL. To set the option, following line is needed in the batch script as comma separated list:

#SBATCH --mail-type=<option>,<option>

User may also specify mail address other than default (system default is the user submitting the job):

#SBATCH --mail-user=<mail@address>

Additionally if array jobs are used, user may wish to receive single notification once the whole array is done (default behaviour). Or by setting ARRAY_TASKS -option, receive mail notification for every task of the array. Latter may be useful for debugging but may be inconvenient if array has thousands of tasks.

#SBATCH --mail-type=ARRAY_TASKS,<other options>

Other resource requests

Below are some of the most common batch job options to specify resource requests.

Job Wall Time limit:

#SBATCH -t <Wall Time limit>

Job CPU count equals to the cores desired:

#SBATCH -c <CPU count>

Job memory limit per core (Please Note: seff or scacct may be used to find out actual memory usage of completed job):

#SBATCH --men-per-cpu=<MB>

Some applications benefit from multithreading which can be set up to 2. Default is 1 task per node. Note that sbatch  does not launch tasks, it requests an allocation of resources and submits a batch script. Example of requesting 2 tasks:

#SBATCH -n 2

A Special case - Job Arrays

Slurm supports job arrays where multiple jobs can be executed with identical parameters. Each job in the array is considered for execution separately, like any other ordinary batch job and is therefore also subject to the fairness calculations.  Ukko2 supports practically unlimited array size and their use is recommended. However, please do pay attention to the special nature of the arrays when constructing your own.

Please note: If using array jobs, and the job requirements change during array run, it is advisable to divide the array to chunks that require same amount of resources. Overestimating resource requirements prohibits other jobs from using reserved resources.

#SBATCH -a  <indexes>

While arrays are extremely efficient way to submit large quantity of jobs, it is not advisable to submit array with entirely different task requirements and base the resource request on the highest array task. (Eg. 2000 1 core tasks using 15Mb of memory and 1000 1 core tasks using 250Gb of memory, and setting the limit to latter for whole 3000 task array. Instead, it is far more efficient to create separate 2000 and 1000 task arrays with appropriate resource requests).

Example of a simple batch file:

#!/bin/bash -l
#SBATCH -J hello_SLURM // name of the batch
#SBATCH -o output.txt  // stdout
#SBATCH -e errors.txt  // stderror
#SBATCH -t 01:20:00    // time reserved for the batch
#SBATCH -c 1           // number of CPU's (Cores) requested
echo "Hello SLURM"     // commands to be run

For more examples, please see the bottom of the CSC's documentation page:
Please Note: CSC instructions are not compatible as-is; ukko does not require queue names except for GPU and test use.

Slurm commands for job control

Now that you have a batch job file, you can run it:

sbatch -options batch_job_file

If you want to cancel your job:

scancel   <jobID>   

To check the status of your jobs that are in the queue

squeue -l -u yourusername

For information about a job that is running

scontrol show <jobID>

For information about a finished job's efficiency. Output of seff is automatically included in the end of job mail notifications, if notifications are set to be sent in the batch script.

seff <jobID>

PBS-like commands for job control

User who is more familiar with PBS, Torque and alike may find following job control wrappers welcomed. 

To submit a batch job (–help provides additional options):

qsub [OPTIONS] [batch script]

To query the queuing system statistics (–help provides additional options):


Change the queued job parameters without removing the job from the queue:

qalter [-N Name] [-r y|n] [-o output file] <jobID>

Setting job into a held state:

qhold [-h u|o|s|n] <jobID>

Release job held with qhold:

qrls [-h u|o|s|a|n] <jobID>

Rerun queued job - eg. place job from running state back to queue:

qrerun [-? | --help] [--man] [--verbose] <jobID>

Delete queued job:

qdel <jobID> 


To be able to deal with numerous software/libraries/compilers and their versions, Ukko 2 uses a module system.
With the module system you can easily customize your working environment to be exactly as you want it.

Modules system modifies the environment variables of the user's shell so that the correct versions of executables are in the path and linker can find the correct version of needed libraries. For example, the command mpicc points to different compilers depending of the loaded module.

Note that not all modules are compatible with each other.

Module system, programs and modules

To list all programs that have been installed:

rpm -qa

To list all available modules that are compatible with the modules that you have currently loaded

module avail

To load a module

module load <modulename>

To switch or swap between two modules

module swap <module1> <module2>

To list all loaded modules

module list

To get info about a module

module help <module-name>

To list all available modules.
If you add the name or part of the name as an argument, you will get more information about matching modules.
If a version number is provided, you also see that module's requirements.

module spider int

To unload a module

module unload <module-name>

To unload all modules

module purge

To create a virtual environment, in this example Python 3.5.2 

module load Python/3.5.2-foss-2016b
mkdir python352 # any directory name, for your virtual environment
virtualenv python352
cd python352
source bin/activate pip install biopython # etc, any python packages you need

You may also install additional packages with pip inside the virtualenv:

./configure --prefix=$HOME/python352
make install

If a module or a program you need is missing, contact the it support

Further instructions

CSC's Taito cluster's documentation is quite useful for Ukko 2 as well.

Check specially the SLURM instructions.

Technical specifications

Operating system: CentOS 7

  • 31 regular computing nodes: ukko2-02- ukko2–32 + 1 login node: ukko2
    •  2 CPUs x 14 cores and 2 threads
      • 2 x  Xeon E5-2680 v4 2.4GHz,35M Cache,9.60GT/s QPI,Turbo,HT,14C/28T (120W)  
    • 256 GB RAM
      • 8 x 32GB RDIMM, 2400MT/s  
    • 1 x 200GB SSD uSATA Mix Use Slim MLC 6Gbps  
    • 1 x 1.8" Backplane with 1 Hard Drive and 1 Infiniband Port
  • 2 big memory nodes: ukko2-paavo and ukko2-pekka
    • 4 CPUs, 48 cores and 2 threads
      • 4 x  Intel Xeon E7-8890 v4 2.2GHz,60M Cache,9.6GT/s QPI 24C/48T,HT,Turbo (165W)  
    • 3 TB RAM
      • 48 x  64GB LRDIMM, 2400MT/s, Quad Rank, x4 Data Width  
    • 2 x 120GB SSD SATA Boot MLC 6Gbps 2.5in Hot-plug Drive  
    • 1 x QLogic 2562, Dual Port 8Gb Optical Fibre Channel HBA 2 SR  
    • 2+2 x Power Supply, Redundant Configuration, 1100W 2 SR  
    • 1 x QLogic 57800 2x10Gb DA/SFP+ + 2x1Gb BT Network Daughter Card 2 SR  
    • 1 x Mellanox ConnectX-4 Single Port, EDR, VPI QSFP28 Full Height Network Adapter 2 S     
  • 2 Tesla P100 GPU nodes: ukko2-g01, ukko2-g02) 
    •  2 CPUs x 14 cores and 2 threads
      • 2 x  Xeon E5-2680 v4 2.4GHz,35M Cache,9.60GT/s QPI,Turbo,HT,14C/28T (120W)  
    • 512 GB RAM
      • 8 x 32GB RDIMM, 2400MT/s  
    • 1 x 200GB SSD uSATA Mix Use Slim MLC 6Gbps  
    • 1 x 1.8" Backplane with 1 Hard Drive and 1 Infiniband Port
    • 4 x nvidia Tesla P100 

  • No labels