We expect the HPC clusters users to know what an HPC cluster is and what parallel computing is. As all HPC clusters are different, it is important for any users to have a general understanding of the clusters they are working on, what they offer and what are their limitations.

This section gives an overview of the technical HPC infrastructure and how things work at the University of Geneva. More details can be found in the corresponding sections of this documentation.

The last part of this page gives more details for advanced users.

The University of Geneva owns three HPC clusters or supercomputers : Baobab, Yggdrasil and Bamboo.

As for now, they are completely separated entities, each of them with their own private network, storage, and login node. What is shared is the accounting (user accounts and job usage).

Choose the right cluster for your work:

• You can use every clusters, but try to stick to one of them.
• Use the cluster where the private partition you have access to is located.
• If you need to access other servers not located in Astro, use Baobab or Bamboo to save bandwidth.
• As your data are stored locally on each cluster, avoid to use both clusters if this involves a lot of data moving between cluster.

You can't submit jobs from one cluster to the other one. This may be done in the future.

Boabab is physically located at Uni Dufour in Geneva downtown, while Yggdrasil is located at the Observatory of Geneva in Sauverny and Bamboo is located in campus Biotech

Each cluster is composed of :

• a login node (aka headnode) allowing users to connect and submit jobs to the cluster.
• many compute nodes which provide the computing power. The compute nodes are not all identical ; they all provide CPU cores (from 8 to 128 cores depending on the model), and some nodes also have GPUs or more RAM (see below).
• management servers that you don't need to worry about, that's the HPC engineers' job. The management servers are here to provide the necessary services such as all the applications (with EasyBuild / module), Slurm job management and queuing system, ways for the HPC engineers to (re-)deploy compute nodes automatically, etc.
• BeeGFS storage servers which provide “fast” parallel file system to store the data from your $HOME and for the scratch data ($HOME/scratch).

All those servers (login, compute, management and storage nodes) :

• run with the GNU/Linux distribution Rocky.
• are inter-connected on high speed InfiniBand network
  • 40Gbit/s (QDR) for Baobab.
  • 100Gbit/s (EDR) for Yggdrasil.
  • 100Gbit/s (EDR) for Bamboo.

In order to provide hundreds of software and versions, we use EasyBuild / module. It allows you to load the exact version of a software/library that is compatible with your code. Learn more about EasyBuild/module

When you want to use some cluster's resources, you need to connect to the login node and submit a job to Slurm which is our job management and queuing system. The job is submitted with an sbatch script (a Bash/shell script with special instructions for Slurm such as how many CPU you need, which Slurm partition to use how long your script will run and how to execute your code). Slurm will place your job in a queue with other users' jobs, and find the fastest way to provide the resources you asked for. When the resources are available, your job will start.
Learn more about Slurm

One important note about Slurm is the concept of partition. When you submit a job, you have to specify a partition that will give you access to some specific resources. For instance, you can submit a job that will use only CPU or GPU nodes.

At the end of September 2024 we sent out a communication about the cost model that will start in January 2025, we'll integrate the important information into our documentation below, but in the meantime you can have a look at the pdf. See the cost section in our FAQ

Overview:

You can find the whole table that you can send to the FNS here.

Users of a given PI are entitled to 100k CPU hours per year free of charge (per PI, not per user). See how to check PI and user past usage.

The cost of renting a compute node is calculated based on the vendor price of the node, adjusted to account for operational and infrastructure expenses. Specifically, we add 15% to the vendor price to cover additional costs, such as maintenance and administrative overhead. The total cost is then amortized over an estimated 5-year lifespan of the compute node to determine the monthly rental rate.

For example, consider a CPU compute node with a vendor price of 14,361 CHF. Adding 15% for extra costs brings the total to 16,515.15 CHF. Dividing this by 60 months (5 years) results in a monthly rental cost of approximately 275.25 CHF.

For more details or to request a specific quote, please contact the HPC support team.

Users are entitled to utilize up to 60% of the computational resources they own or rent within the cluster. For example, if you rent a compute node with 128 CPU cores for one year, you will receive a total credit of 128 (cores) × 24 (hours) × 365 (days) × 0.6 (max usage rate) = 672,768 core-hours. This credit can be used across any of our three clusters – Bamboo, Baobab, and Yggdrasil – regardless of where the compute node was rented or purchased.

The main advantage is that you are not restricted to using your private nodes, but can access the three clusters and even the GPUs.

We are developing scripts to allow to check the usage and the amount of hours you have the right to use regarding the hardware your group owns.

The key distinction when using your own resources is that you benefit from a higher scheduling priority, ensuring quicker access to computational resources. As well, you are not restricted to using your private nodes, but can access the three clusters and even the GPUs.

For more details, please contact the HPC support team.

Research groups have the option to purchase or rent “private” compute nodes to expand the resources available in our clusters. This arrangement provides the group with a private partition, granting higher priority access to the specified nodes (resulting in reduced wait times) and extended job runtimes of up to 7 days (compared to 4 days for public compute nodes).

• Shared Integration: The compute node is added to the corresponding shared partition. Other users may utilize it when the owning group is not using it. For details, refer to the partitions section.
• Maximum Usage: Research groups can utilize up to 60% of the node's maximum theoretical computational capacity. This ensures fair access to shared resources. See Usage limit
• Cost: In addition to the base cost of the compute node, a 15% surcharge is applied to cover operational expenses such as cables, racks, switches, and storage.
• Ownership Period: The compute node remains the property of the research group for 5 years. After this period, the node may remain in production but will only be accessible via public and shared partitions.
• Warranty and Repairs: Nodes come with a 3-year warranty. If the node fails after this period, the research group is responsible for 100% of repair costs. Repairing the node involves sending it to the vendor for diagnostics and a quote, with a maximum diagnostic fee of 420 CHF, even if the node is irreparable.
• Administrative Access: The research group does not have administrative rights over the node.
• Maintenance: The HPC team handles the installation and maintenance of the compute node, ensuring it operates consistently with other nodes in the cluster.
• Decommissioning: The HPC team may decommission the node if it becomes obsolete, but it will remain in production for at least 5 years.

See below the current price of a compute node (without the extra 15% and without VAT)

• 2 x 64 Core AMD EPYC 7742 2.25GHz Processor
• 512GB DDR4 3200MHz Memory (16x32GB)
• 100G IB EDR card
• 960GB SSD
• ~ 14'442.55 CHF TTC

• 2 x 96 Core AMD EPYC 9754 2.4GHz Processor
• 768GB DDR45 4800MHz Memory (24x32GB)
• 100G IB EDR card
• 960GB SSD
• ~ 16'464 CHF TTC

Key differences:

• + 9754 has higher memory performance of up to 460.8 GB/s vs 7763 which has 190.73 GB/s
• + 9754 has a bigger cache
• - 9754 is more expensive
• - power consumption is 400W for 9754 vs 240W for 7763
• - 9754 is more difficult to cool as the inlet temperature for air colling must be 22° max

• 2 x 64 Core AMD EPYC 9554 3.15GHz Processor
• 768GB DDR4 4800MHz ECC Server Memory (24x 32GB / 0 free slots)
• 1 x 7.68TB NVMe Intel 24×7 Datacenter SSD (14PB written until warranty end)
• 4 x nVidia Ampere H100 94GB PCIe GPU (max. 8 GPUs possible)
• ~ 124k CHF HT
• ~ 28,5k CHF HT per extra GPU

• 2 x 64 Core AMD EPYC 9554 3.10GHz Processor
• 384 GB DDR4 4800MHz ECC Server Memory
• 8 x nVidia RTX 4090 24GB Graphics Controller
• ~ 44k CHF HT

We usually install and order the nodes twice per year.

If you want to ask a financial contribution from UNIGE you must complete a COINF application : https://www.unige.ch/rectorat/commissions/coinf/appel-a-projets

Baobab, our HPC infrastructure, supports educators in providing students with hands-on HPC experience.

Teachers can request access via [dw.unige.ch](final link to be added later, use hpc@unige.ch in the meantime), and once the request is fulfilled, a special account named <PI_NAME>_teach will be created for the instructor. Students must specify this account when submitting jobs for course-related work (e.g., --account=<PI_NAME>_teach).

A shared storage space can also be created optionally, accessible at /home/share/<PI_NAME>_teach and/or /srv/beegfs/scratch/shares/<PI_NAME>_teach.

All student usage is free of charge if they submit their job to the correct account.

We strongly recommend that teachers use and promote our user-friendly web portal at OpenOndDemand which supports tools like Matlab, JupyterLab, and more. Baobab helps integrate real-world computational tools into curricula, fostering deeper learning in HPC technologies.

Everyone has different needs for their computation. A typical example of usage of the cluster would consists of these steps :

• connect to the login node
• this will give you access to the data from your $HOME directory
• execute an sbatch script which will request resources to Slurm for the estimated runtime (i.e. : 16 CPU cores, 8 GB RAM for up to 7h on partition “shared-cpu”). The sbatch will contain instructions/commands :
  • for Slurm scheduler to access compute resources for a certain time
  • to load the right application and libraries with module for your code to work
  • on how to execute your application.
• the Slurm job will be placed in the Slurm queue
• once the requested resources are available, your job will start and be executed on one or multiple compute nodes (which can all access the BeeGFS $HOME and $HOME/scratch directories).
• all communication and data transfer between the nodes, the storage and the login node go through the InfiniBand network.

If you want to know what type of CPU and architecture is supported, check the section For Advanced users.

Both clusters contain a mix of “public” nodes provided by the University of Geneva, a “private” nodes in general paid 50% by the University and 50% by a research group for instance. Any user of the clusters can request compute resources on any node (public and private), but a research group who owns “private” nodes has a higher priority on its “private” nodes and can request a longer execution time.

We have several GPU models on the cluster. You can find here a table of what is available.

On Baobab

If more than one GPU model can be selected if you didn't specify a constraint, they are allocated in the the same order as they are listed in the table. The low end GPU first (GPU with a lower weight are selected first).

On Yggdrasil

When you request a GPU, you can either specify no model at all or you can give specific constraints such as double precision.

See here how to request GPU for your jobs.

Since our clusters are regularly expanded, the nodes are not all from the same generation. You can see the details in the following table.

The “generation” column is just a way to classify the nodes on our clusters. In the following table you can see the features of each architecture.

In the following table you can see which type of GPU is available on Baobab.

Link to see the GPU details https://developer.nvidia.com/cuda-gpus#compute

Since our clusters are regularly expanded, the nodes are not all from the same generation. You can see the details in the following table.

The “generation” column is just a way to classify the nodes on our clusters. In the following table you can see the features of each architecture.

Click here to Compare Intel CPUS.

In the following table you can see which type of GPU is available on Yggdrasil.

Link to see the GPU details https://developer.nvidia.com/cuda-gpus#compute

In order to follow system ressources, you can go to https://monitor.hpc.unige.ch/dashboards

You can reach node metrics for the last 30 days and BeeGFS metrics for the last 6 months.

For checking resources on a specific node, go to “Baobab - General” or “Yggdrasil - General” and click on “Host Overview - Single”. You will be able to choose the node you want to check in the form at the top :

For going back to the dashboard list, click on the four squares on the left panel :

The “General” dashboard in “Yggdrasil - General” and “Baobab - General” folders gives an overview of the cluster : total load and memory used, and how many nodes are up/down.

You can see GPU metrics too under “Cuda - GPU” dashboards.