User Tools

Site Tools


hpc:hpc_clusters

How our clusters work

We expect the HPC clusters users to know what an HPC cluster is and what parallel computing is. As all HPC clusters are different, it is important for any users to have a general understanding of the clusters they are working on, what they offer and what are their limitations.

This section gives an overview of the technical HPC infrastructure and how things work at the University of Geneva. More details can be found in the corresponding sections of this documentation.

The last part of this page gives more details for advanced users.

The clusters : Baobab and Yggdrasil

The University of Geneva owns two HPC clusters or supercomputers : Baobab and Yggdrasil.

As for now, they are two completely separated entities, each of them with their own private network, storage, and login node. What is shared is the accounting (user accounts and job usage).

Choose the right cluster for your work:

  • You can use both clusters, but try to stick to one of them.
  • Use the cluster where the private partition you have access to is located.
  • If you need to access other servers not located in Astro, use Baobab to save bandwidth.
  • As your data are stored locally on each cluster, avoid to use both clusters if this involves a lot of data moving between cluster.
  • Use Yggdrasil if you need newer CPUs, compute nodes with up to 1.5TB of memory, volta GPU.

You can't submit jobs from one cluster to the other one. This may be done in the future.

Boabab is physically located at Uni Dufour in Geneva downtown, while Yggdrasil is located at the Observatory of Geneva in Sauverny.

cluster name datacentre Interconnect public CPU public GPU Total CPU size Total GPU size
Baobab Dufour IB 40GB QDR ~900 0 ~9'736 271
Yggdrasil Astro IB 100GB EDR ~3000 44 ~8'228 52

How do our clusters work ?

Overview

Each cluster is composed of :

  • a login node (aka headnode) allowing users to connect and submit jobs to the cluster.
  • many compute nodes which provide the computing power. The compute nodes are not all identical ; they all provide CPU cores (from 8 to 128 cores depending on the model), and some nodes also have GPUs or more RAM (see below).
  • management servers that you don't need to worry about, that's the HPC engineers' job. The management servers are here to provide the necessary services such as all the applications (with EasyBuild / module), Slurm job management and queuing system, ways for the HPC engineers to (re-)deploy compute nodes automatically, etc.
  • BeeGFS storage servers which provide “fast” parallel file system to store the data from your $HOME and for the scratch data ($HOME/scratch).

All those servers (login, compute, management and storage nodes) :

  • run with the GNU/Linux distribution Rocky.
  • are inter-connected on high speed InfiniBand network
    • 40Gbit/s (QDR) for Baobab.
    • 100Gbit/s (EDR) for Yggdrasil.

In order to provide hundreds of software and versions, we use EasyBuild / module. It allows you to load the exact version of a software/library that is compatible with your code. Learn more about EasyBuild/module

When you want to use some cluster's resources, you need to connect to the login node and submit a job to Slurm which is our job management and queuing system. The job is submitted with an sbatch script (a Bash/shell script with special instructions for Slurm such as how many CPU you need, which Slurm partition to use how long your script will run and how to execute your code). Slurm will place your job in a queue with other users' jobs, and find the fastest way to provide the resources you asked for. When the resources are available, your job will start.
Learn more about Slurm

One important note about Slurm is the concept of partition. When you submit a job, you have to specify a partition that will give you access to some specific resources. For instance, you can submit a job that will use only CPU or GPU nodes.

Private nodes

Important update, draft preview. We are currently in the process of implementing changes to the investment approach for the HPC service Baobab, wherein research groups will no longer purchase physical nodes as their property. Instead, they will have the option to pay for a share and duration of usage. This new approach offers several advantages for both the research groups and us as the service provider.

For research groups, the main advantage lies in the increased flexibility it provides. They can now tailor their investments to suit the specific needs of their projects, scaling their usage as required. This eliminates the constraints of owning physical nodes and allows for more efficient allocation of resources.

As the service provider, we benefit from this new investment model as well. We can now strategically purchase compute nodes and hardware based on the actual demand from research groups, ensuring that our investments align with their usage patterns. This allows us to optimize resource utilization and make timely acquisitions when needed.

In cases where research groups have already purchased compute nodes, we offer them the opportunity to convert their ownership into credits for shares. We estimate that a compute node typically lasts for at least 6 years under normal conditions, and this conversion option ensures that the value of their existing investment is not lost.

Research groups can buy “private” nodes to add in our clusters, which means their research group has a private partition with a higher priority to use those specific nodes (less waiting time) and they can run their jobs for a longer time (7 days instead of 4 for public compute nodes).

Rules:

  • The compute node remains the research group property
  • There is a three-year warranty on the compute node. If there is a failure after the warranty period, 100% of the repair costs will be the responsibility of the research group. If the node is out of order, you have the option to have it repaired. In order to get a quote, we need to send the compute node to the vendor and the initial cost they will charge to do a quick diagnostic and make a quote is a maximum of 420 CHF, even if the node can't be repaired (worst case).
  • The research groups hasn't an admin right on it
  • The compute node is installed and maintained by the HPC team in the same way as the other compute nodes
  • The HPC team can decide to decommission the node when it is too old but the hardware will be in production for at least four years

See the partitions section to have more details about the integration of your private node in the cluster.

Current price of a compute node is:

– AMD –

  • 2 x 64 Core AMD EPYC 7742 2.25GHz Processor
  • 512GB DDR4 3200MHz Memory (16x32GB)
  • 100G IB EDR card
  • 960GB SSD
  • ~ 14'442.55 CHF TTC

– GPU A100 with AMD –

  • 1 x 64 Core AMD EPYC 7742 2.25GHz Processor
  • 256GB DDR4 3200MHz ECC Server Memory (8x 32GB / 0 free slots)
  • 1 x 1.92TB SATAIII Intel 24×7 Datacenter SSD (6.5PB written until warranty end)
  • 1 x nVidia Tesla A100 80GB PCIe GPU passive cooled (max. 4 GPUs possible)
  • ~ 24'300 CHF TTC
  • ~ 11'270 CHF TTC per extra GPU

– GPU RTX3090 with AMD –

  • 2 x 64 Core AMD EPYC 7742 2.25GHz Processor
  • 512GB DDR4 3200MHz ECC Server Memory
  • 8 x nVidia RTX 3090 24GB Graphics Controller
  • ~ 42'479 CHF TTC

We usually install and order the nodes twice per year.

If you want to ask a financial contribution from UNIGE you must complete a COINF application : https://www.unige.ch/rectorat/commissions/coinf/appel-a-projets

How do I use your clusters ?

Everyone has different needs for their computation. A typical example of usage of the cluster would consists of these steps :

  • connect to the login node
  • this will give you access to the data from your $HOME directory
  • execute an sbatch script which will request resources to Slurm for the estimated runtime (i.e. : 16 CPU cores, 8 GB RAM for up to 7h on partition “shared-cpu”). The sbatch will contain instructions/commands :
    • for Slurm scheduler to access compute resources for a certain time
    • to load the right application and libraries with module for your code to work
    • on how to execute your application.
  • the Slurm job will be placed in the Slurm queue
  • once the requested resources are available, your job will start and be executed on one or multiple compute nodes (which can all access the BeeGFS $HOME and $HOME/scratch directories).
  • all communication and data transfer between the nodes, the storage and the login node go through the InfiniBand network.

If you want to know what type of CPU and architecture is supported, check the section For Advanced users.

For advanced users

Infrastructure schema

FIXME

Compute nodes

Both clusters contain a mix of “public” nodes provided by the University of Geneva, a “private” nodes in general paid 50% by the University and 50% by a research group for instance. Any user of the clusters can request compute resources on any node (public and private), but a research group who owns “private” nodes has a higher priority on its “private” nodes and can request a longer execution time.

GPUs models on the clusters

We have several GPU models on the cluster. You can find here a table of what is available.

On Baobab

Model Memory GRES Constraint gpu arch Compute Capability minimum CUDA version Precision Weight
Titan X 12GB titan COMPUTE_TYPE_TITAN COMPUTE_CAPABILITY_6_1 8.0 SIMPLE_PRECISION_GPU 10
P100 12GB pascal COMPUTE_TYPE_PASCAL COMPUTE_CAPABILITY_6_0 8.0 DOUBLE_PRECISION_GPU 20
RTX 2080 Ti 11GB turing COMPUTE_TYPE_TURING COMPUTE_CAPABILITY_7_5 10.0 SIMPLE_PRECISION_GPU 30
RTX 3080 10GB ampere COMPUTE_TYPE_AMPERE COMPUTE_CAPABILITY_8_6 11.0 SIMPLE_PRECISION_GPU 40
RTX 3090 25GB ampere COMPUTE_TYPE_AMPERE COMPUTE_CAPABILITY_8_6 11.0 SIMPLE_PRECISION_GPU 50
A5000 25GB ampere COMPUTE_TYPE_AMPERE COMPUTE_CAPABILITY_8_6 11.0 SIMPLE_PRECISION_GPU 50
A100 40GB ampere COMPUTE_TYPE_AMPERE COMPUTE_CAPABILITY_8_0 11.0 DOUBLE_PRECISION_GPU 60
A100 80GB ampere COMPUTE_TYPE_AMPERE COMPUTE_CAPABILITY_8_0 11.0 DOUBLE_PRECISION_GPU 70

If more than one GPU model can be selected if you didn't specify a constraint, they are allocated in the the same order as they are listed in the table. The low end GPU first (GPU with a lower weight are selected first).

On Yggdrasil

Model Memory GRES Constraint gpu arch Compute Capability Precision Weight
Titan RTX 24GB turing COMPUTE_TYPE_TURING COMPUTE_CAPABILITY_7.5 SIMPLE_PRECISION_GPU 10
V100 32GB volta COMPUTE_TYPE_VOLTA COMPUTE_CAPABILITY_7.0 DOUBLE_PRECISION_GPU 20

When you request a GPU, you can either specify no model at all or you can give specific constraints such as double precision.

If you are doing machine learning for example, you DON'T need double precision. Double precision is useful for software doing for example physical numerical simulations.
We don't have mixed GPUs models on the same node. Every GPU node has only one GPU model.

See here how to request GPU for your jobs.

Bamboo (coming soon)

Generation Model Freq Nb cores Architecture Nodes Memory Extra flag Status
V8 EPYC-7742 2.25GHz 128 cores “Rome” (7 nm) node[001-043] 512GB to be installed
V8 EPYC-72F3 3.7GHz 16 cores “Rome” (7 nm) node[044-045] 1TB BIG_MEM to be installed
GPU model Architecture Mem Compute Capability Slurm resource Nb per node Nodes Peer access between GPUs
RTX 3090 Ampere 25GB 8.6 ampere 8 gpu[001,002] NO
A100 Ampere 80GB 8.0 amper 4 gpu[003] YES

Baobab

CPUs on Baobab

Since our clusters are regularly expanded, the nodes are not all from the same generation. You can see the details in the following table.

Generation Model Freq Nb cores Architecture Nodes Extra flag Status
V2 X5650 2.67GHz 12 cores “Westmere-EP” (32 nm) node[093-101,103-111,140-153 decommissioned
V3 E5-2660V0 2.20GHz 16 cores “Sandy Bridge-EP” (32 nm) node[009-010,012-018,020-025,029-044] decommissioned in 2023
V3 E5-2660V0 2.20GHz 16 cores “Sandy Bridge-EP” (32 nm) node[001-005,007-008,011,019,026-028,045-056,058] to be decommissioned in 2022
V3 E5-2670V0 2.60GHz 16 cores “Sandy Bridge-EP” (32 nm) node[059,061-062] to be decommissioned in 2022
V3 E5-4640V0 2.40GHz 32 cores “Sandy Bridge-EP” (32 nm) node[186] to be decommissioned in 2022
V4 E5-2650V2 2.60GHz 16 cores “Ivy Bridge-EP” (22 nm) node[063-066,154-172] to be decommissioned in 2022
V5 E5-2643V3 3.40GHz 12 cores “Haswell-EP” (22 nm) gpu[002,012] on prod
V6 E5-2630V4 2.20GHz 20 cores “Broadwell-EP” (14 nm) node[173-185,187-201,205-213] on prod
gpu[004-010] on prod
V6 E5-2637V4 3.50GHz 8 cores “Broadwell-EP” (14 nm) node[218-219] HIGH_FREQUENCY on prod
V6 E5-2643V4 3.40GHz 12 cores “Broadwell-EP” (14 nm) node[202,204,216-217] HIGH_FREQUENCY on prod
V6 E5-2680V4 2.40GHz 28 cores “Broadwell-EP” (14 nm) node[203] on prod
V7 EPYC-7601 2.20GHz 64 cores “Naples” (14 nm) gpu[011] on prod
V8 EPYC-7742 2.25GHz 128 cores “Rome” (7 nm) node[273-277,285-288,312-335] gpu[013-031] on prod
V9 GOLD-6240 2.60GHz 36 cores “Cascade Lake” (14 nm) node[265-272] on prod

The “generation” column is just a way to classify the nodes on our clusters. In the following table you can see the features of each architecture.

MMXSSESSE2SSE3SSSE3SSE4.1SSE4.2AVXF16CAVX2FMA3NB AVX-512 FMA
Westmere-EP YESYESYES YES YES YES YES NO NO NO NO
Sandy Bridge-EPYESYESYES YES YES YES YES YESNO NO NO
Ivy Bridge-EP YESYESYES YES YES YES YES YESYES NO NO
Haswell-EP YESYESYES YES YES YES YES YESYES YES NO
Broadwell-EP YESYESYES YES YES YES YES YESYES YES YES
Naples YESYESYES YES YES YES YES YESYES YES YES
Rome YESYESYES YES YES YES YES YESYES YES YES
Cascade Lake YESYESYES YES YES YES YES YESYES YES YES 2

GPUs on Baobab

In the following table you can see which type of GPU is available on Baobab.

GPU model Architecture Mem Compute CapabilitySlurm resourceNb per node Nodes
Titan X Pascal 12GB 6.1 titan 6 gpu[002]
P100 Pascal 12GB 6.0 pascal 6 gpu[004]
P100 Pascal 12GB 6.0 pascal 5 gpu[005]
P100 Pascal 12GB 6.0 pascal 8 gpu[006]
P100 Pascal 12GB 6.0 pascal 4 gpu[007]
Titan X Pascal 12GB 6.1 titan 8 gpu[008-010]
RTX 2080 Ti Turing 11GB 7.5 turing 2 gpu[011]
RTX 2080 Ti Turing 11GB 7.5 turing 8 gpu[012-016]
RTX 2080 Ti Turing 11GB 7.5 turing 4 gpu[018-019]
RTX 3090 Ampere 25GB 8.6 ampere 8 gpu[017,021,025-026,034-035]
RTX A5000 Ampere 25GB 8.6 ampere 8 gpu[044]
RTX 3080 Ampere 10GB 8.6 ampere 8 gpu[023-024,036-43]
A100 Ampere 40GB 8.0 ampere 2 gpu[027]
A100 Ampere 40GB 8.0 ampere 6 gpu[022]
A100 Ampere 40GB 8.0 ampere 1 gpu[028]
A100 Ampere 40GB 8.0 ampere 4 gpu[020,030-031]
A100 Ampere 80GB 8.0 ampere 4 gpu[029]
A100 Ampere 80GB 8.0 ampere 3 gpu[032-033]
A100 Ampere 80GB 8.0 ampere 2 gpu[045]

Link to see the GPU details https://developer.nvidia.com/cuda-gpus#compute

Yggdrasil

CPUs on Yggdrasil

Since our clusters are regularly expanded, the nodes are not all from the same generation. You can see the details in the following table.

Generation Model Freq Nb cores Architecture Nodes Extra flag
V9 GOLD-6240 2.60GHz 36 cores “Intel Xeon Gold 6240 CPU @ 2.60GHz” cpu[001-023,024-111,120-122]
V9 GOLD-6244 3.60GHz 16 cores “Intel Xeon Gold 6244 CPU @ 3.60GHz” cpu[112-115]
V8 EPYC-7742 2.25GHz 128 cores “AMD EPYC 7742 64-Core Processor” cpu[116-119,123-150]
V9 SILVER-4208 2.10GHz 16 cores “Intel Xeon Silver 4208 CPU @ 2.10GHz” gpu[001-006,008]
V9 GOLD-6234 3.30GHz 16 cores “Intel Xeon Gold 6234 CPU @ 3.30GHz” gpu[007]

The “generation” column is just a way to classify the nodes on our clusters. In the following table you can see the features of each architecture.

SSE4.2 AVX AVX2 NB AVX-512 FMA
Intel Xeon Gold 6244 YES YES YES 2
Intel Xeon Gold 6240 YES YES YES 2
Intel Xeon Gold 6234 YES YES YES 2
Intel Xeon Silver 4208 YES YES YES 1

Click here to Compare Intel CPUS.

GPUs on Yggdrasil

In the following table you can see which type of GPU is available on Yggdrasil.

GPU model Architecture Mem Compute Capability Slurm resource Nb per node Nodes Peer access between GPUs
Titan RTX Turing 24GB 7.5 turing 8 gpu[001,002,004] NO
Titan RTX Turing 24GB 7.5 turing 6 gpu[003,005] NO
Titan RTX Turing 24GB 7.5 turing 4 gpu[006,007] NO
V100 Volta 32GB 7.0 volta 1 gpu[008] YES

Link to see the GPU details https://developer.nvidia.com/cuda-gpus#compute

Monitoring performance

In order to follow system ressources, you can go to https://monitor.hpc.unige.ch/dashboards

You can reach node metrics for the last 30 days and BeeGFS metrics for the last 6 months.

For checking resources on a specific node, go to “Baobab - General” or “Yggdrasil - General” and click on “Host Overview - Single”. You will be able to choose the node you want to check in the form at the top :

For going back to the dashboard list, click on the four squares on the left panel :

The “General” dashboard in “Yggdrasil - General” and “Baobab - General” folders gives an overview of the cluster : total load and memory used, and how many nodes are up/down.

You can see GPU metrics too under “Cuda - GPU” dashboards.

Job accounting

If you are interested by your HPC usage, group usage, wait time, etc. we have the right tool for you: Open XDMoD.

We track job usage of our clusters here: https://openxdmod.hpc.unige.ch/

If you want to have more details, you need to login. This instance isn't integrated with our SI right now. In order to get an account, you need to clik on “sign in” top left of the page and then on “Don't have an account?”.

hpc/hpc_clusters.txt · Last modified: 2024/02/12 14:59 by Yann Sagon