Differences

This shows you the differences between two versions of the page.

--- hpc:best_practices [2022/12/06 11:03] – [Transfer data from cluster to another] Adrien Albert
+++ hpc:best_practices [2024/10/07 08:15] (current) – [HPC cluster user departure procedure] Adrien Albert
@@ Line 1: / Line 1: @@
-<title> Best practices and smart use of the HPC resources </title>
+{{METATOC 1-5}}
-This page gives best practices and tips on how to use the clusters **Baobab** and **Yggdrasil**.
 ====== Introduction ======
+This page gives best practices and tips on how to use the clusters **Baobab** and **Yggdrasil**.
 An HPC cluster is an advanced, complex and always-evolving piece of technology. It's easy to forget details and make mistakes when using one, so don't hesitate to check this section every now and then, yes, even if you are the local HPC guru in your team! There's always something new to learn !
@@ Line 14: / Line 15: @@
 For your first steps we recommend the following :
   * Check the [[hpc:best_practices#rules_and_etiquette|Rules and etiquette]].
-  * Connect to the login node of the cluster you are planning to use :
+  * Connect to the login node of the cluster you are planning to use : [[hpc:access_the_hpc_clusters|Access : SSH, X2GO]]
-    * [[hpc:access_the_hpc_clusters|Access : SSH, X2GO]]
   * Check the rest of this page for best practices and smart use of the HPC resources.
     * [[hpc:best_practices#stop_wasting_resources|This page contains important information]]! You can hog resources and/or wait much longer than you could if you don't request the right amount of time for your job, if you ask too much (or not enough) resources, if you are not using the right partition, etc.
-  * Understand how to load your libraries/application with ''module''
+  * Understand how to load your libraries/application with ''module'': [[applications_and_libraries|Applications and libraries]]
-    * [[applications_and_libraries|Applications and libraries]]
+  * Learn how to write a Slurm ''sbatch'' script: [[slurm|Slurm and job management]]
-  * Learn how to write a Slurm ''sbatch'' script
-    * [[slurm|Slurm and job management]]
 ====== Rules and etiquette ======
@@ Line 159: / Line 157: @@
 ) Go to your directory containing ''<nowiki>the_best_project_ever</nowiki>'':
 <code>
-cd $HOME/my_projects/
+(baobab)-[toto@login2 ~]$cd $HOME/my_projects/
 </code>
 ) Set the variables (or not)
 <code>
-DST=$HOME/my_projects/
+(baobab)-[toto@login2 my_projects]$ DST=$HOME/my_projects/
-DIR=the_best_project_ever
+(baobab)-[toto@login2 my_projects]$ DIR=the_best_project_ever
-YGGDRASIL=login1.yggdrasil
+(baobab)-[toto@login2 my_projects]$ YGGDRASIL=login1.yggdrasil
 </code>
 ) Run the rsync
 <code>
-rsync -abviuzPrg $DIR $YGGDRASIL:$DST
+(baobab)-[toto@login2 my_projects]$ rsync -aviuzPrg ${DIR} ${YGGDRASIL}:${DST}
 </code>
+====== SWITCHfilesender from the cluster ======
+Switch Filesender
+Filesender is a service provided by SWITCH for transferring files via http. Normally, files are downloaded via a web browser, but using the CLI is more efficient and we benefit from the Login node's high bandwidth.
+In order to avoid having to transfer the files to your local computer it is possible to use the Filesender command line tools as explained below
+===== Configuring the CLI tools =====
+Connect to https://filesender.switch.ch then go to the profile tab
+{{:hpc:screenshot-2022-01-13-at-15-14-02.png}}
+Then click on "Create API secret" to generate a code that will be used to allow you to authenticate
+{{:hpc:screenshot-2022-01-13-at-15-14-37.png}}
+This will generate a long string like
+''ab56bf28434d1fba1d5f6g3aaf8776e55fd722df205197''
+Then connect to HPC cluster and run the following commands to download the CLI tool and the configuration
+<code>
+cd
+mkdir ~/.filesender
+wget https://filesender.switch.ch/clidownload.php -O filesender.py
+wget https://filesender.switch.ch/clidownload.php?config=1 -O ~/.filesender/filesender.py.ini
+</code>
+You will then need to edit the ''~/.filesender/filesender.py.ini'' file using your preferred tool
+You need to enter your username as show in the Filesender profile and the API key that you generated
+Note that at present, unlike the other Switch services this is not your EduID account!
+<code>
+[system]
+base_url = https://filesender.switch.ch/filesender2/rest.php
+default_transfer_days_valid = 20
+[user]
+username = Charizard@unige.ch
+apikey = ab56bf28434d1fba1d5f6g3aaf8776e55fd722df205197
+</code>
+==== Transferring files ====
+Now that we have done this we can transfer files - note that the modules must be loaded in order to have a python with the required libraries.
+  (baobab)-[alberta@login2 ~]$ module load GCC Python
+You may need to install some python package before:
+  (baobab)-[alberta@login2 ~]$ pip3 install requests urllib3
+<code>
+(baobab)-[alberta@login2 ~]$ python3 filesender.py -p -r mewtwo@unige.ch master.tar.gz
+Uploading: /home/users/a/alberta/master.tar.gz 0-18408 0%
+Uploading: /home/users/a/alberta/master.tar.gz 18408 100%
+</code>
+A mail will be sent to ''Mewtwo@unige.ch''  who can then download the file
+====== HPC cluster user departure procedure ======
+**Step 1: Backup and clean up personal data**
+  * **Retrieve your personal data** from the **home** and **scratch** directories.
+  * **Clean** these directories by removing unnecessary files.
+**Step 2 (optional): Organizing and transferring shared data**
+  * If you belong to a group, **identify important data** in shared directories.
+  * If a data ownership transfer is necessary, send a **request to the HPC team** to transfer ownership to another user, with the group leader's approval.
+**Step 3: Transfer confirmation**
+  * Ensure the data transfer has been completed and notify the relevant parties.