hpc:storage_on_hpc
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| hpc:storage_on_hpc [2025/03/14 10:40] – [NASAC] Gaël Rossignol | hpc:storage_on_hpc [2026/05/12 15:30] (current) – [Temporary shared space] Yann Sagon | ||
|---|---|---|---|
| Line 12: | Line 12: | ||
| This is the storage space we offer on our clusters | This is the storage space we offer on our clusters | ||
| - | ^ Cluster | + | ^ Cluster |
| - | | Baobab | + | | Baobab |
| - | | ::: | ''/ | + | | ::: | ''/ |
| - | | ::: | ''/ | + | | ::: | ''/ |
| - | | Yggdrasil | ''/ | + | | Yggdrasil | ''/ |
| - | | ::: | ''/ | + | | ::: | ''/ |
| + | | Bamboo | ||
| + | | ::: | ''/ | ||
| We realize you all have different needs in terms of storage. To guarantee storage space for all users, we have **set a quota on home and scratch directory**, | We realize you all have different needs in terms of storage. To guarantee storage space for all users, we have **set a quota on home and scratch directory**, | ||
| Line 157: | Line 159: | ||
| ===== Temporary shared space ===== | ===== Temporary shared space ===== | ||
| If you need to access the data from more than one node, you can use a space reachable from all your jobs running on the same compute node. When you have no more jobs running on the node, the content of the storage is erased. | If you need to access the data from more than one node, you can use a space reachable from all your jobs running on the same compute node. When you have no more jobs running on the node, the content of the storage is erased. | ||
| + | <WRAP center round important 60%> | ||
| + | The new path for shared space is ''/ | ||
| + | </ | ||
| - | The path is the following: ''/ | + | |
| + | The path is the following: '' | ||
| See here for a usage example: https:// | See here for a usage example: https:// | ||
| ====== Sharing files with other users ====== | ====== Sharing files with other users ====== | ||
| - | Sometimes you need to share files with some colleagues or your research group. | + | Sometimes, you may need to share files with colleagues or members of your research group. |
| + | |||
| + | We offer two types of shared folders: | ||
| - | We provide two types of shared folders : | + | * **In the " |
| - | | + | * **In the " |
| - | * in " | + | |
| - | If you need one, please fill the form on DW: https:// | + | To request a shared folder, please fill out the form at [[https:// |
| - | If you are an Outisder user and you don't have access | + | A **group** is a collection of users used to manage shared access to resources. These groups |
| + | If you need more details about groups, | ||
| + | If you are an *Outsider* user and do not have access to DW, please ask your **PI** to submit the request on your behalf. | ||
| <note important> | <note important> | ||
| You are not allowed to change the permission of your '' | You are not allowed to change the permission of your '' | ||
| Line 230: | Line 239: | ||
| <code console> | <code console> | ||
| - | (baobab)-[sagon@login2 | + | (baobab)-[sagon@login1 |
| home dir: /home/sagon | home dir: /home/sagon | ||
| scratch dir: / | scratch dir: / | ||
| Line 312: | Line 321: | ||
| <code console> | <code console> | ||
| [sagon@login1 ~] $ dbus-launch bash | [sagon@login1 ~] $ dbus-launch bash | ||
| + | </ | ||
| + | |||
| + | **If you are using sbatch add a sleep after '' | ||
| + | |||
| + | < | ||
| + | dbus-launch bash | ||
| + | sleep 3 | ||
| + | gio mount .... | ||
| </ | </ | ||
| Line 336: | Line 353: | ||
| </ | </ | ||
| - | <note important> | + | <note important> |
| - | If you need to access the data on the nodes, you need to mount them there as well in your sbatch script.</ | + | If you need to access the on other nodes, you need to mount them there as well in your sbatch script.</ |
| If you need to script this, you can put your credentials in a file in your home directory. | If you need to script this, you can put your credentials in a file in your home directory. | ||
| Line 380: | Line 397: | ||
| reference: (([[https:// | reference: (([[https:// | ||
| + | |||
| + | === Sometimes mount is not available but you can browse/ | ||
| + | |||
| + | < | ||
| + | $ dbus-launch bash | ||
| + | |||
| + | $ gio mount smb:// | ||
| + | Authentication Required | ||
| + | Enter user and password for share “hpc_exchange” on “nasac-evs2.unige.ch”: | ||
| + | User [rossigng]: s-hpc-share | ||
| + | Domain [SAMBA]: ISIS | ||
| + | Password: | ||
| + | |||
| + | $ gio mount -l | ||
| + | Drive(0): SAMSUNG MZ7L3480HBLT-00A07 | ||
| + | Type: GProxyDrive (GProxyVolumeMonitorUDisks2) | ||
| + | Drive(1): SAMSUNG MZ7L3480HBLT-00A07 | ||
| + | Type: GProxyDrive (GProxyVolumeMonitorUDisks2) | ||
| + | Mount(0): hpc_exchange on nasac-evs2.unige.ch -> smb:// | ||
| + | Type: GDaemonMount | ||
| + | |||
| + | $ gio list smb:// | ||
| + | backup | ||
| + | |||
| + | $ gio list smb:// | ||
| + | toto | ||
| + | titi | ||
| + | tata.txt | ||
| + | |||
| + | $ gio cp smb:// | ||
| + | |||
| + | ... | ||
| + | </ | ||
| + | |||
| ===== CVMFS ===== | ===== CVMFS ===== | ||
| All the compute nodes of our clusters have CernVM-FS client installed. CernVM-FS, the CernVM File System (also known as CVMFS), is a file distribution service that is particularly well suited to distribute software installations across a large number of systems world-wide in an efficient way. | All the compute nodes of our clusters have CernVM-FS client installed. CernVM-FS, the CernVM File System (also known as CVMFS), is a file distribution service that is particularly well suited to distribute software installations across a large number of systems world-wide in an efficient way. | ||
| Line 413: | Line 464: | ||
| The EESSI did a nice tutorial about CVMFS readable on [[https:// | The EESSI did a nice tutorial about CVMFS readable on [[https:// | ||
| + | ===== EOS ===== | ||
| + | You can mount root filesystems using EOS. | ||
| + | < | ||
| + | (bamboo)-[sagon@login1 ~]$ export EOS_MGM_URL=root:// | ||
| + | (bamboo)-[sagon@login1 ~]$ export EOS_HOME=/ | ||
| + | (bamboo)-[sagon@login1 ~]$ eos fuse mount / | ||
| + | </ | ||
| + | <note important> | ||
| ====== Robinhood ====== | ====== Robinhood ====== | ||
| Robinhood Policy Engine is a versatile tool to manage contents of large file systems. It daily scans the scratch beegfs filesystems. It makes it possible to schedule mass action on filesystem entries by defining attribute-based policies. | Robinhood Policy Engine is a versatile tool to manage contents of large file systems. It daily scans the scratch beegfs filesystems. It makes it possible to schedule mass action on filesystem entries by defining attribute-based policies. | ||
hpc/storage_on_hpc.1741948848.txt.gz · Last modified: (external edit)