eResearch Doc

User Tools

Site Tools

You are here: start » hpc » applications_and_libraries
hpc:applications_and_libraries

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
hpc:applications_and_libraries [2025/01/15 09:39] – [FOSS toolchain] Yann Sagonhpc:applications_and_libraries [2025/11/04 09:48] (current) – [Custom Python lib] Yann Sagon
Line 313: Line 313:
  
 ===== Conda =====  ===== Conda ===== 
 +==== How to Create a Conda Environment in a Container =====
  
-Use it 
  
-<code> +Using **Conda** directly on HPC systems or shared servers can cause performance issues and storage overload because Conda environments create thousands of small files. This often results in: 
-module load Anaconda3+ 
 +  * Slow job startup times 
 +  * Filesystem limitations being hit 
 +  * High I/O load on the cluster 
 +  * Complex environment management 
 + 
 +A better solution is to **encapsulate Conda environments inside a container**. This way, the entire environment is packaged into a single file (such as a `.sif` image used by Apptainer/Singularity), which can be easily deployed, shared, and reused without polluting clusters. 
 + 
 + 
 +=== Benefits === 
 +Using this method offers multiple advantages: 
 +  - ✅ **Fewer files**: Your environment is stored in a single `.sif` file 
 +  - ✅ **Portability**: Share the container easily with collaborators 
 +  - ✅ **Reproducibility**: Your environment stays consistent across systems 
 +  - ✅ **Isolation**: No risk of interfering with other environment 
 +  - ✅ **Stability**: Cluster updates will not break your environment 
 + 
 +=== Limitations === 
 +  - ⚠️ The container is static; to update packages, you need to rebuild the image 
 + 
 + 
 +This guide explains how to build such a container using [[https://hpc-community.unige.ch/t/new-software-installed-cotainr-version-2025-3-0/3917|cotainr]], a tool that simplifies container creation. 
 + 
 + 
 +=== Step 1 – Define the Conda Environment === 
 +Create a file ''env.yml'' that contains the definition of your environment: 
 +(As exemple we will use ''bioenv.yml'') 
 + 
 +<code bash> 
 +name: bioenv 
 +channels: 
 +  - bioconda 
 +  - conda-forge 
 +  - defaults 
 +dependencies: 
 +  - blast=2.16.0 
 +  - diamond=2.1.11 
 +  - exonerate=2.4.0 
 +  - spades=4.1.0 
 +  - mafft=7.525 
 +  - trimal=1.5.0 
 +  - numpy 
 +  - joblib 
 +  - scipy 
 +[...] 
 + 
 +prefix:/home/users/a/alberta/env     # <==== To delete/edit
 </code> </code>
 +
 +You can generate this file using the following commands:
 +
 +<code bash>
 +# 1. (optional) create your environment (or not if you already have one)
 +$ conda create -n bioenv -c bioconda -c conda-forge spades exonerate diamond blast mafft trimal numpy joblib scipy -y
 +
 +# 2. Activate your environment
 +$ conda activate bioenv
 +
 +# 3. Export the settings of your environment 
 +# It’s recommended to manually remove the `prefix:` line at the bottom of the file before using it with cotainr.
 +$ conda env export > bioenv.yml
 +</code>
 +
 +
 +
 +=== Step 2 – Build the Container ===
 +Now use ''cotainr'' to create the image:
 +
 +<code bash>
 +$ module load GCCcore/13.3.0 cotainr
 +# Ex: cotainr build <env.sif> --base-image=docker://<WhatEverYouWant>:latest --accept-licenses --conda-env=<env.yml>
 +$ cotainr build bioenv.sif --base-image=docker://ubuntu:latest --accept-licenses --conda-env=bioenv.yml
 +</code>
 +
 +You can replace ''ubuntu:latest'' with any other base image, such as ''rockylinux:latest''.
 +
 +=== Step 3 – Use the Container ===
 +You can now run commands inside the container as follows:
 +
 +<code bash>
 +
 +$ apptainer exec bioenv.sif python3 -c "import numpy; print(numpy.__version__)"
 +</code>
 +
 +Or launch any program inside the container just like you would in a normal environment.
 +
 +
 +
 +
 +
  
  
 ==== Conda environment management ==== ==== Conda environment management ====
  
 +
 +Use it
 +
 +<code>
 +module load Anaconda3
 +</code>
  
 Create Create
Line 997: Line 1091:
 ==== Custom Python lib ==== ==== Custom Python lib ====
  
-If you need to install a python library or a different version of the ones already installed, virtualenv is the solution.+If you need to install a Python library or a different version of the ones already installed, **virtualenv** is the solution.
  
-Python-virtualenv is installed on Baobab http://www.virtualenv.org/en/latest/+Python-virtualenv is installed on Baobab: http://www.virtualenv.org/en/latest/
  
-Begin by loading a version of python using module (see above)+Begin by loading a version of Python using the module system (see above).
  
-Create a new virtualenv if it's not already existing (put it where you want and name it like you want):+Before creating your virtual environment, load the required modules.   
 +You must load **Python** and any additional Python packages you need (for example, SciPy-bundle):
  
-For ''virtualenv'' 20 or above (starting from ''Python/3.8.2'' ): <code console> +<code> 
-capello@login2:~$ virtualenv ~/baobab_python_env+ml GCC/14.3.0 Python/3.11.6 virtualenv/20.32.0 SciPy-bundle/2025.07
 </code> </code>
  
-For previous ''virtualenv'' versions: <code console> +Create a new virtual environment (you can choose the location and name).   
-capello@login2:~$ virtualenv --no-site-packages ~/baobab_python_env+<WRAP center round tip 60%> 
 +It is recommended to use the option ''<nowiki>--system-site-packages</nowiki>'' so that you can benefit from the Python modules provided by the loaded modules. 
 +</WRAP> 
 + 
 +<code> 
 +virtualenv --system-site-packages ~/baobab_python_env
 </code> </code>
  
-This will create a directory named baobab_python_env in your home directory.+This will create a directory named **baobab_python_env** in your home directory.
  
-Every time you want to use your virtualenv, you should activate it first:+Every time you want to use your virtual environment, you should activate it first:
  
 <code> <code>
 . ~/baobab_python_env/bin/activate . ~/baobab_python_env/bin/activate
 </code> </code>
 +
 +<WRAP center round important 60%>
 +Before reusing this virtual environment, you must reload the same modules you used when creating it (e.g., Python and SciPy-bundle). Otherwise, some packages may not be available.
 +</WRAP>
 +
  
 Install all the needed packages in the environment: Install all the needed packages in the environment:
Line 1032: Line 1137:
 ~/baobab_python_env/bin/python ~/baobab_python_env/bin/python
 </code> </code>
- 
 ==== Pip install from source ==== ==== Pip install from source ====
 By default when you use ''pip'' to install a library, it will download a binary of ''.whl'' file instead of building the module from source. This may be an issue if the module itself depend on a custom ''libc'' version or is optimized for a given kind of CPU. In this case, you can force ''pip'' to install a module by building it from source. By default when you use ''pip'' to install a library, it will download a binary of ''.whl'' file instead of building the module from source. This may be an issue if the module itself depend on a custom ''libc'' version or is optimized for a given kind of CPU. In this case, you can force ''pip'' to install a module by building it from source.
Line 1056: Line 1160:
 With the Baobab upgrade to CentOS 7 (cf. https://hpc-community.unige.ch/t/baobab-migration-from-centos6-to-centos7/361 ) we do not provide anymore a central RStudio. With the Baobab upgrade to CentOS 7 (cf. https://hpc-community.unige.ch/t/baobab-migration-from-centos6-to-centos7/361 ) we do not provide anymore a central RStudio.
  
-Instead, you can download the upstream Open Source binary RStudio Desktop version (cf. https://rstudio.com/products/rstudio/download/ ) and directly use it, here the instructions: +Instead, we provide Rstudio on [[hpc:how_to_use_openondemand|OpenOnDemand]].
- +
-  - install it in your ''${HOME}'' folder: <code console> +
-capello@login2:~$ mkdir Downloads +
-capello@login2:~$ cd Downloads +
-capello@login2:~/Downloads$ wget ${URL_FOR_rstudio-${VERSION}-x86_64-fedora.tar.gz} +
-[...] +
-capello@login2:~/Downloads$ tar axvf rstudio-${VERSION}-x86_64-fedora.tar.gz +
-[...] +
-capello@login2:~/Downloads$  +
-</code> +
-  - launch an interactive graphical job: +
-    - connect to the cluster using [[hpc:access_the_hpc_clusters#gui_accessdesktop_with_x2go|GUI access / Desktop with X2Go]] or using ''ssh -Y'' from a machine with an X server such as [[hpc:access_the_hpc_clusters#from_linux_and_mac_os|Linux or Mac]]. \\  +
-    - start an interactive session on a node (see [[hpc/slurm#interactive_jobs|Interactive Slurm jobs]]): <code console> +
-capello@login2:~$ salloc -p debug-cpu -n 1 -c 16 --x11 +
-salloc: Pending job allocation 39085914 +
-salloc: job 39085914 queued and waiting for resources +
-salloc: job 39085914 has been allocated resources +
-salloc: Granted job allocation 39085914 +
-capello@node001:~$  +
-</code> Doing so, you will have 16 cores on one node of the partition ''debug-cpu'' for a max time of 15 minutes. Specify the appropriate duration time, partition, etc. like you would do for a normal job. +
-    - load one of the R version supported by RStudio, for example:<code console> +
-capello@node001:~$ module spider R/3.6.0 +
- +
----------------------------------------------------------------------------------- +
-  R: R/3.6.0 +
----------------------------------------------------------------------------------- +
-    Description: +
-      R is a free software environment for statistical computing and +
-      graphics.  +
- +
- +
-    You will need to load all module(s) on any one of the lines below +
-    before the "R/3.6.0" module is available to load. +
- +
-      GCC/8.2.0-2.31.1  OpenMPI/3.1.3 +
-[...] +
-capello@node001:~$ module load GCC/8.2.0-2.31.1  OpenMPI/3.1.3 +
-capello@node001:~$ module load PostgreSQL/11.3-Python-3.7.2 +
-capello@node001:~$ module load R/3.6.0 +
-capello@node001:~$  +
-</code> +
-    - run RStudio : <code console> +
-capello@node001:~$ ~/Downloads/rstudio-${VERSION}/bin/rstudio +
-</code> +
- +
-<note important>Latest version of Rstudio needs an aditional dependency loaded +
-<code> +
-module load PostgreSQL/11.3-Python-3.7.2 +
-</code> +
-</note> +
 ==== R packages ==== ==== R packages ====
  
hpc/applications_and_libraries.1736933987.txt.gz · Last modified: (external edit)