We expect the HPC clusters users to know what an HPC cluster is and what parallel computing is. As all HPC clusters are different, it is important for any users to have a general understanding of the clusters they are working on, what they offer and what are their limitations.

This section gives an overview of the technical HPC infrastructure and how things work at the University of Geneva. More details can be found in the corresponding sections of this documentation.

The last part of this page gives more details for advanced users.

The University of Geneva owns three HPC clusters or supercomputers : Baobab, Yggdrasil and Bamboo.

As for now, they are completely separated entities, each of them with their own private network, storage, and login node. What is shared is the accounting (user accounts and job usage).

Choose the right cluster for your work:

• You can use every clusters, but try to stick to one of them.
• Use the cluster where the private partition you have access to is located.
• If you need to access other servers not located in Astro, use Baobab or Bamboo to save bandwidth.
• As your data are stored locally on each cluster, avoid to use both clusters if this involves a lot of data moving between cluster.

You can't submit jobs from one cluster to the other one. This may be done in the future.

Boabab is physically located at Uni Dufour in Geneva downtown, while Yggdrasil is located at the Observatory of Geneva in Sauverny and Bamboo is located in campus Biotech

Each cluster is composed of :

• a login node (aka headnode) allowing users to connect and submit jobs to the cluster.
• many compute nodes which provide the computing power. The compute nodes are not all identical ; they all provide CPU cores (from 8 to 128 cores depending on the model), and some nodes also have GPUs or more RAM (see below).
• management servers that you don't need to worry about, that's the HPC engineers' job. The management servers are here to provide the necessary services such as all the applications (with EasyBuild / module), Slurm job management and queuing system, ways for the HPC engineers to (re-)deploy compute nodes automatically, etc.
• BeeGFS storage servers which provide “fast” parallel file system to store the data from your $HOME and for the scratch data ($HOME/scratch).

All those servers (login, compute, management and storage nodes) :

• run with the GNU/Linux distribution Rocky.
• are inter-connected on high speed InfiniBand network
  • 40Gbit/s (QDR) for Baobab.
  • 100Gbit/s (EDR) for Yggdrasil.
  • 100Gbit/s (EDR) for Bamboo.

In order to provide hundreds of software and versions, we use EasyBuild / module. It allows you to load the exact version of a software/library that is compatible with your code. Learn more about EasyBuild/module

When you want to use some cluster's resources, you need to connect to the login node and submit a job to Slurm which is our job management and queuing system. The job is submitted with an sbatch script (a Bash/shell script with special instructions for Slurm such as how many CPU you need, which Slurm partition to use how long your script will run and how to execute your code). Slurm will place your job in a queue with other users' jobs, and find the fastest way to provide the resources you asked for. When the resources are available, your job will start.
Learn more about Slurm

One important note about Slurm is the concept of partition. When you submit a job, you have to specify a partition that will give you access to some specific resources. For instance, you can submit a job that will use only CPU or GPU nodes.

At the end of September 2024 we sent out a communication about the cost model that will start in January 2025, we'll integrate the important information into our documentation below, but in the meantime you can have a look at the pdf. See the cost section in our FAQ

Overview:

You can find the whole table that you can send to the FNS here.

Research groups can buy “private” nodes to add to our clusters, which means that their research group has a private partition with a higher priority to use these specific nodes (less waiting time) and they can run their jobs for a longer time (7 days instead of 4 for public compute nodes).

Rules:

• The compute node is added to the corresponding shared partition, which means that other users can use it when it is not being used by its owner. See partitions section to have more details about the integration of your private node in the cluster.
• In addition to the regular cost of the compute node (see below for an example), we add 15% to the price to cover additional operational costs such as cables, racks, switches, storage, etc.
• The compute node remains the property of the research group for a period of 5 years. After this time, the node can remain in production but will only be available via public and shared partitions.
• There is a three year warranty on the compute node. If the node fails after the warranty period, the research group will be responsible for 100% of the repair costs. If the node fails, you have the option to have it repaired. In order to get a quote, we'll need to send the compute node to the vendor, and the initial cost they'll charge to do a quick diagnostic and make a quote is a maximum of 420 CHF, even if the node can't be repaired (worst case).
• The research group doesn't have administrative rights on it.
• The compute node is installed and maintained by the HPC team in the same way as the other compute nodes.
• The HPC team can decide to decommission the node when it is too old, but the hardware will be in production for at least four years.

Please note that you may as well rent private nodes for a minimal duration of 6 months instead of buying it.

See below the current price of a compute node (without the extra 15% and without VAT)

• 2 x 64 Core AMD EPYC 7742 2.25GHz Processor
• 512GB DDR4 3200MHz Memory (16x32GB)
• 100G IB EDR card
• 960GB SSD
• ~ 14'442.55 CHF TTC

• 2 x 64 Core AMD EPYC 9554 3.15GHz Processor
• 768GB DDR4 4800MHz ECC Server Memory (24x 32GB / 0 free slots)
• 1 x 7.68TB NVMe Intel 24×7 Datacenter SSD (14PB written until warranty end)
• 4 x nVidia Ampere H100 94GB PCIe GPU (max. 8 GPUs possible)
• ~ 124k CHF HT
• ~ 28,5k CHF HT per extra GPU

• 2 x 64 Core AMD EPYC 9554 3.10GHz Processor
• 384 GB DDR4 4800MHz ECC Server Memory
• 8 x nVidia RTX 4090 24GB Graphics Controller
• ~ 44k CHF HT

We usually install and order the nodes twice per year.

If you want to ask a financial contribution from UNIGE you must complete a COINF application : https://www.unige.ch/rectorat/commissions/coinf/appel-a-projets

Everyone has different needs for their computation. A typical example of usage of the cluster would consists of these steps :

• connect to the login node
• this will give you access to the data from your $HOME directory
• execute an sbatch script which will request resources to Slurm for the estimated runtime (i.e. : 16 CPU cores, 8 GB RAM for up to 7h on partition “shared-cpu”). The sbatch will contain instructions/commands :
  • for Slurm scheduler to access compute resources for a certain time
  • to load the right application and libraries with module for your code to work
  • on how to execute your application.
• the Slurm job will be placed in the Slurm queue
• once the requested resources are available, your job will start and be executed on one or multiple compute nodes (which can all access the BeeGFS $HOME and $HOME/scratch directories).
• all communication and data transfer between the nodes, the storage and the login node go through the InfiniBand network.

If you want to know what type of CPU and architecture is supported, check the section For Advanced users.

Both clusters contain a mix of “public” nodes provided by the University of Geneva, a “private” nodes in general paid 50% by the University and 50% by a research group for instance. Any user of the clusters can request compute resources on any node (public and private), but a research group who owns “private” nodes has a higher priority on its “private” nodes and can request a longer execution time.

We have several GPU models on the cluster. You can find here a table of what is available.

On Baobab

If more than one GPU model can be selected if you didn't specify a constraint, they are allocated in the the same order as they are listed in the table. The low end GPU first (GPU with a lower weight are selected first).

On Yggdrasil

When you request a GPU, you can either specify no model at all or you can give specific constraints such as double precision.

See here how to request GPU for your jobs.

Since our clusters are regularly expanded, the nodes are not all from the same generation. You can see the details in the following table.

The “generation” column is just a way to classify the nodes on our clusters. In the following table you can see the features of each architecture.

In the following table you can see which type of GPU is available on Baobab.

Link to see the GPU details https://developer.nvidia.com/cuda-gpus#compute

Since our clusters are regularly expanded, the nodes are not all from the same generation. You can see the details in the following table.

The “generation” column is just a way to classify the nodes on our clusters. In the following table you can see the features of each architecture.

Click here to Compare Intel CPUS.

In the following table you can see which type of GPU is available on Yggdrasil.

Link to see the GPU details https://developer.nvidia.com/cuda-gpus#compute

In order to follow system ressources, you can go to https://monitor.hpc.unige.ch/dashboards

You can reach node metrics for the last 30 days and BeeGFS metrics for the last 6 months.

For checking resources on a specific node, go to “Baobab - General” or “Yggdrasil - General” and click on “Host Overview - Single”. You will be able to choose the node you want to check in the form at the top :

For going back to the dashboard list, click on the four squares on the left panel :

The “General” dashboard in “Yggdrasil - General” and “Baobab - General” folders gives an overview of the cluster : total load and memory used, and how many nodes are up/down.

You can see GPU metrics too under “Cuda - GPU” dashboards.