eResearch Doc

User Tools

Site Tools

You are here: start » hpc » applications_and_libraries
hpc:applications_and_libraries

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
hpc:applications_and_libraries [2025/04/11 09:20] – [How to Create a Conda Environment in a Container] Adrien Alberthpc:applications_and_libraries [2025/11/04 09:48] (current) – [Custom Python lib] Yann Sagon
Line 338: Line 338:
  
  
-This guide explains how to build such a container using [`cotainr`](https://hpc-community.unige.ch/t/new-software-installed-cotainr-version-2025-3-0/3917), a tool that simplifies container creation.+This guide explains how to build such a container using [[https://hpc-community.unige.ch/t/new-software-installed-cotainr-version-2025-3-0/3917|cotainr]], a tool that simplifies container creation.
  
  
 === Step 1 – Define the Conda Environment === === Step 1 – Define the Conda Environment ===
-Create a file `env.yml` that contains the definition of your environment: +Create a file ''env.yml'' that contains the definition of your environment: 
-(As exemple we will use `bioenv.yml`)+(As exemple we will use ''bioenv.yml'')
  
 <code bash> <code bash>
Line 363: Line 363:
 [...] [...]
  
-prefix:/home/users/a/alberta/env+prefix:/home/users/a/alberta/env     # <==== To delete/edit
 </code> </code>
  
Line 371: Line 371:
 # 1. (optional) create your environment (or not if you already have one) # 1. (optional) create your environment (or not if you already have one)
 $ conda create -n bioenv -c bioconda -c conda-forge spades exonerate diamond blast mafft trimal numpy joblib scipy -y $ conda create -n bioenv -c bioconda -c conda-forge spades exonerate diamond blast mafft trimal numpy joblib scipy -y
 +
 # 2. Activate your environment # 2. Activate your environment
 $ conda activate bioenv $ conda activate bioenv
 +
 # 3. Export the settings of your environment  # 3. Export the settings of your environment 
-#    It’s recommended to manually remove the `prefix:` line at the bottom of the file before using it with cotainr.+# It’s recommended to manually remove the `prefix:` line at the bottom of the file before using it with cotainr.
 $ conda env export > bioenv.yml $ conda env export > bioenv.yml
 </code> </code>
  
-<note tip></note>+
  
 === Step 2 – Build the Container === === Step 2 – Build the Container ===
-Now use `cotainr` to create the image:+Now use ''cotainr'' to create the image:
  
 <code bash> <code bash>
Line 389: Line 391:
 </code> </code>
  
-You can replace `ubuntu:latest` with any other base image, such as `rockylinux:latest`.+You can replace ''ubuntu:latest'' with any other base image, such as ''rockylinux:latest''.
  
 === Step 3 – Use the Container === === Step 3 – Use the Container ===
Line 403: Line 405:
  
  
-=== References === + 
-  * [cotainr tutorial](https://hpc-community.unige.ch/t/tutorial-use-conda-in-a-container/3529) + 
-  * [Apptainer documentation](https://apptainer.org/docs/)+ 
 + 
 +==== Conda environment management ====
  
  
Line 413: Line 417:
 module load Anaconda3 module load Anaconda3
 </code> </code>
- 
- 
-==== Conda environment management ==== 
- 
  
 Create Create
Line 1091: Line 1091:
 ==== Custom Python lib ==== ==== Custom Python lib ====
  
-If you need to install a python library or a different version of the ones already installed, virtualenv is the solution.+If you need to install a Python library or a different version of the ones already installed, **virtualenv** is the solution.
  
-Python-virtualenv is installed on Baobab http://www.virtualenv.org/en/latest/+Python-virtualenv is installed on Baobab: http://www.virtualenv.org/en/latest/
  
-Begin by loading a version of python using module (see above)+Begin by loading a version of Python using the module system (see above).
  
-Create a new virtualenv if it's not already existing (put it where you want and name it like you want):+Before creating your virtual environment, load the required modules.   
 +You must load **Python** and any additional Python packages you need (for example, SciPy-bundle):
  
-For ''virtualenv'' 20 or above (starting from ''Python/3.8.2'' ): <code console> +<code> 
-capello@login2:~$ virtualenv ~/baobab_python_env+ml GCC/14.3.0 Python/3.11.6 virtualenv/20.32.0 SciPy-bundle/2025.07
 </code> </code>
  
-For previous ''virtualenv'' versions: <code console> +Create a new virtual environment (you can choose the location and name).   
-capello@login2:~$ virtualenv --no-site-packages ~/baobab_python_env+<WRAP center round tip 60%> 
 +It is recommended to use the option ''<nowiki>--system-site-packages</nowiki>'' so that you can benefit from the Python modules provided by the loaded modules. 
 +</WRAP> 
 + 
 +<code> 
 +virtualenv --system-site-packages ~/baobab_python_env
 </code> </code>
  
-This will create a directory named baobab_python_env in your home directory.+This will create a directory named **baobab_python_env** in your home directory.
  
-Every time you want to use your virtualenv, you should activate it first:+Every time you want to use your virtual environment, you should activate it first:
  
 <code> <code>
 . ~/baobab_python_env/bin/activate . ~/baobab_python_env/bin/activate
 </code> </code>
 +
 +<WRAP center round important 60%>
 +Before reusing this virtual environment, you must reload the same modules you used when creating it (e.g., Python and SciPy-bundle). Otherwise, some packages may not be available.
 +</WRAP>
 +
  
 Install all the needed packages in the environment: Install all the needed packages in the environment:
Line 1126: Line 1137:
 ~/baobab_python_env/bin/python ~/baobab_python_env/bin/python
 </code> </code>
- 
 ==== Pip install from source ==== ==== Pip install from source ====
 By default when you use ''pip'' to install a library, it will download a binary of ''.whl'' file instead of building the module from source. This may be an issue if the module itself depend on a custom ''libc'' version or is optimized for a given kind of CPU. In this case, you can force ''pip'' to install a module by building it from source. By default when you use ''pip'' to install a library, it will download a binary of ''.whl'' file instead of building the module from source. This may be an issue if the module itself depend on a custom ''libc'' version or is optimized for a given kind of CPU. In this case, you can force ''pip'' to install a module by building it from source.
Line 1150: Line 1160:
 With the Baobab upgrade to CentOS 7 (cf. https://hpc-community.unige.ch/t/baobab-migration-from-centos6-to-centos7/361 ) we do not provide anymore a central RStudio. With the Baobab upgrade to CentOS 7 (cf. https://hpc-community.unige.ch/t/baobab-migration-from-centos6-to-centos7/361 ) we do not provide anymore a central RStudio.
  
-Instead, you can download the upstream Open Source binary RStudio Desktop version (cf. https://rstudio.com/products/rstudio/download/ ) and directly use it, here the instructions: +Instead, we provide Rstudio on [[hpc:how_to_use_openondemand|OpenOnDemand]].
- +
-  - install it in your ''${HOME}'' folder: <code console> +
-capello@login2:~$ mkdir Downloads +
-capello@login2:~$ cd Downloads +
-capello@login2:~/Downloads$ wget ${URL_FOR_rstudio-${VERSION}-x86_64-fedora.tar.gz} +
-[...] +
-capello@login2:~/Downloads$ tar axvf rstudio-${VERSION}-x86_64-fedora.tar.gz +
-[...] +
-capello@login2:~/Downloads$  +
-</code> +
-  - launch an interactive graphical job: +
-    - connect to the cluster using [[hpc:access_the_hpc_clusters#gui_accessdesktop_with_x2go|GUI access / Desktop with X2Go]] or using ''ssh -Y'' from a machine with an X server such as [[hpc:access_the_hpc_clusters#from_linux_and_mac_os|Linux or Mac]]. \\  +
-    - start an interactive session on a node (see [[hpc/slurm#interactive_jobs|Interactive Slurm jobs]]): <code console> +
-capello@login2:~$ salloc -p debug-cpu -n 1 -c 16 --x11 +
-salloc: Pending job allocation 39085914 +
-salloc: job 39085914 queued and waiting for resources +
-salloc: job 39085914 has been allocated resources +
-salloc: Granted job allocation 39085914 +
-capello@node001:~$  +
-</code> Doing so, you will have 16 cores on one node of the partition ''debug-cpu'' for a max time of 15 minutes. Specify the appropriate duration time, partition, etc. like you would do for a normal job. +
-    - load one of the R version supported by RStudio, for example:<code console> +
-capello@node001:~$ module spider R/3.6.0 +
- +
----------------------------------------------------------------------------------- +
-  R: R/3.6.0 +
----------------------------------------------------------------------------------- +
-    Description: +
-      R is a free software environment for statistical computing and +
-      graphics.  +
- +
- +
-    You will need to load all module(s) on any one of the lines below +
-    before the "R/3.6.0" module is available to load. +
- +
-      GCC/8.2.0-2.31.1  OpenMPI/3.1.3 +
-[...] +
-capello@node001:~$ module load GCC/8.2.0-2.31.1  OpenMPI/3.1.3 +
-capello@node001:~$ module load PostgreSQL/11.3-Python-3.7.2 +
-capello@node001:~$ module load R/3.6.0 +
-capello@node001:~$  +
-</code> +
-    - run RStudio : <code console> +
-capello@node001:~$ ~/Downloads/rstudio-${VERSION}/bin/rstudio +
-</code> +
- +
-<note important>Latest version of Rstudio needs an aditional dependency loaded +
-<code> +
-module load PostgreSQL/11.3-Python-3.7.2 +
-</code> +
-</note> +
 ==== R packages ==== ==== R packages ====
  
hpc/applications_and_libraries.1744363229.txt.gz · Last modified: (external edit)