This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
hpc:storage_on_hpc [2022/11/28 12:42] Adrien Albert |
hpc:storage_on_hpc [2024/05/02 13:51] (current) Gaël Rossignol [Cluster storage] |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | < | + | {{METATOC 1-5}} |
+ | ======= | ||
There are different types of storage on the clusters. This is important to understand where to store which type of data. | There are different types of storage on the clusters. This is important to understand where to store which type of data. | ||
Line 11: | Line 12: | ||
This is the storage space we offer on our clusters | This is the storage space we offer on our clusters | ||
- | ^ Cluster | + | ^ Cluster |
- | | Baobab | + | | Baobab |
- | | ::: | ''/ | + | | ::: | ''/ |
- | | Yggdrasil | ''/ | + | | ::: | ''/ |
- | | ::: | ''/ | + | | Yggdrasil | ''/ |
+ | | ::: | ''/ | ||
- | We realize you all have different needs in terms of storage. To guarantee storage space for all users, we have set a quota of 1 TB per user on home beegfs, beyond | + | We realize you all have different needs in terms of storage. To guarantee storage space for all users, we have **set a quota on home and scratch directory**, see table above for details. Beyond |
Line 66: | Line 68: | ||
Also, the scratch directory is not a permanent storage solution, we strongly advise you to move/clean useless/ | Also, the scratch directory is not a permanent storage solution, we strongly advise you to move/clean useless/ | ||
+ | |||
+ | ==== Quota ==== | ||
+ | |||
+ | |||
+ | As the storage is shared by everyone, this ensure a fair scratch usage and prevent users from filling it. We setup a quota based on the number of files you own, not the file size. | ||
+ | |||
+ | **The maximum file count is currently set to 10M.** | ||
+ | |||
+ | What does it mean for you: if the number of files in your scratch space is higher than 10M, you won’t be able to write to it anymore. | ||
+ | |||
+ | Error message: | ||
+ | |||
+ | Disk quota exceeded | ||
+ | |||
+ | To resume the situation, you should clean up some data in your scratch directory. | ||
+ | |||
+ | ===== Fast directory ===== | ||
+ | |||
+ | A new fast storage is available dedicated for jobs using multiples nodes and scratchlocal need to be shared between nodes. | ||
+ | |||
+ | ^ Cluster | ||
+ | | Baobab | ||
+ | |||
+ | <note important> | ||
+ | |||
+ | ==== Quota ==== | ||
+ | |||
+ | |||
+ | As the storage is shared by everyone, this ensure a fair scratch usage and prevent users from filling it. We setup a quota based on the total size. | ||
+ | |||
+ | You should clean up some data in your fast directory as soon as your jobs are finished. | ||
====== Local storage ====== | ====== Local storage ====== | ||
Line 84: | Line 117: | ||
</ | </ | ||
- | ===== temporary | + | ===== Temporary private |
On **each** compute node, you can use the following private ephemeral spaces: | On **each** compute node, you can use the following private ephemeral spaces: | ||
Line 93: | Line 126: | ||
Those places are private and only accessible by your job. | Those places are private and only accessible by your job. | ||
+ | ===== Temporary shared space ===== | ||
+ | If you need to access the data from more than one node, you can use a space reachable from all your jobs running on the same compute node. When you have no more jobs running on the node, the content of the storage is erased. | ||
+ | The path is the following: ''/ | ||
+ | See here for a usage example: https:// | ||
====== Sharing files with other users ====== | ====== Sharing files with other users ====== | ||
Line 117: | Line 154: | ||
For easy sharing you need to set '' | For easy sharing you need to set '' | ||
- | This is a side-effect of the default permissions on Red Hat-based systems without **User Private | + | This is a side-effect of the default permissions on Red Hat-based systems without **User Private |
+ | </ | ||
+ | <note info> | ||
+ | Since we use ACL to set the user right, you can't rely on sticky bit to force the new files to belong to a group which is not your primary group. You have the following options: | ||
+ | * You can request to change your primary group: every file that you create on the cluster will belong to this group | ||
+ | * You can set your umask to 0002 as explained previously | ||
+ | * You can launch on a regular basis as script that " | ||
</ | </ | ||
====== Best practices ====== | ====== Best practices ====== | ||
Line 153: | Line 196: | ||
===== Check disk usage on the clusters ===== | ===== Check disk usage on the clusters ===== | ||
- | Since ''/ | + | ==== Check disk usage on home and scratch ==== |
+ | |||
+ | |||
+ | Since ''/ | ||
The script '' | The script '' | ||
<code console> | <code console> | ||
- | [brero@login2 ~]$ beegfs-get-quota-home-scratch.sh | + | (baobab)-[sagon@login2 ~]$ beegfs-get-quota-home-scratch.sh |
- | USER > /home | + | home dir: /home/sagon |
- | brero > 1.04 GiB | | + | scratch dir: / |
+ | |||
+ | user/ | ||
+ | storage | ||
+ | ----------------------------|------||------------|------------||---------|--------- | ||
+ | home | sagon|240477|| | ||
+ | scratch | ||
</ | </ | ||
- | N.B. This includes all your data in '' | + | <WRAP center round tip 60%> |
+ | This includes all your data in '' | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | |||
+ | ==== Check disk usage on NASAC ==== | ||
If you have space as well in ''/ | If you have space as well in ''/ | ||
Line 294: | Line 352: | ||
reference: (([[https:// | reference: (([[https:// | ||
- | ===== CVMFS ===== | + | ===== CVMFS ===== |
- | The following cvmfs content | + | All the compute nodes of our clusters have CernVM-FS client installed. CernVM-FS, the CernVM File System (also known as CVMFS), is a file distribution service that is particularly well suited to distribute software installations across a large number of systems world-wide in an efficient way. |
+ | |||
+ | |||
+ | |||
+ | A couple of repository are mounted on the compute and login node such as: | ||
* atlas.cern.ch | * atlas.cern.ch | ||
Line 303: | Line 365: | ||
* grid.cern.ch | * grid.cern.ch | ||
- | The content is mounted using autofs. It means that the root directory | + | |
+ | The content is mounted using autofs | ||
didn't access explicitly one of the child directory. Doing so will mount the repository for a couple of | didn't access explicitly one of the child directory. Doing so will mount the repository for a couple of | ||
minutes and unmount it automatically. | minutes and unmount it automatically. | ||
+ | |||
+ | Other flaghship repository available without further configuration: | ||
+ | |||
+ | * unpacked.cern.ch | ||
+ | * singularity.opensciencegrid.org (container registry) | ||
+ | * software.eessi.io ( | ||
< | < | ||
Line 314: | Line 383: | ||
cvmfs-config.cern.ch | cvmfs-config.cern.ch | ||
</ | </ | ||
+ | |||
+ | The EESSI did a nice tutorial about CVMFS readable on [[https:// | ||
+ | |||
====== Robinhood ====== | ====== Robinhood ====== |