Python and modules

The default python installed in our Debian 11 system is python3.9. As everyone use a different combination of python modules and versions, there is only a small amount (python3 -m pip list) of python modules directly installed on the system.

python3 -m pip list | awk '{print "  * "$0}'

You will have to install, in your $HOME or team storage (see Where to store files?), your own environment(s).

venv : Virtual Environment

mylogin@x5570comp2:~/$ mkdir -p ~/tests/py39 && cd ~/tests/py39
mylogin@x5570comp2:~/tests/py39$ python3.9 -m venv ~/tests/py39

This create a minimal python3.9 virtual environment (or venv). You can now install every additionnal python modules you’ll need.

Note

Only if you want to use the already pre-installed python modules, add --system-site-packages when creating your venv. You should’nt.

  • Activate (enter) a virtual environment:

mylogin@x5570comp2:~/tests/py39$ source ~/tests/py39/bin/activate
(py39) mylogin@x5570comp2:~/tests/py39$

  • Exit (deactivate) a virtual environment:

(py39) mylogin@x5570comp2:~/tests/py39$ deactivate
mylogin@x5570comp2:~/tests/py39$

  • Install a python modules (inside a venv)

For example, scipy:

mylogin@x5570comp2:~$ cd tests/py39/
mylogin@x5570comp2:~/tests/py39$ source ~/tests/py39/bin/activate
(py39) mylogin@x5570comp2:~/tests/py39$ python3.9 -m pip install scipy

[... some logs ...]

Installing collected packages: numpy, scipy
Successfully installed numpy-1.22.1 scipy-1.7.3
(py39) mylogin@x5570comp2:~/tests/py39$

  • Using ATLAS (Automatically Tuned Linear Algebra Software) library

For some python modules (numpy, for instance), you may need to add this line to your ~/.bashrc (outside of virtual environment, and source it after editing):

export LD_LIBRARY_PATH="/usr/lib/atlas-base:$LD_LIBRARY_PATH"

then you can install (or upgrade, with -U) numpy, inside your venv:

mylogin@x5570comp2:~$ cd tests/py39/
mylogin@x5570comp2:~/tests/py39$ source ~/tests/py39/bin/activate
(py39) mylogin@x5570comp2:~/tests/py39$ python3.9 -m pip install numpy
Collecting numpy
  Using cached numpy-1.22.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Installing collected packages: numpy
Successfully installed numpy-1.22.1
(py39) mylogin@x5570comp2:~/tests/py39$

You can now use numpy in your virtual environment:

(py39) mylogin@x5570comp2:~/tests/py39$ python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>

Note

With ATLAS library, numpy will be multithread-compatible, hence you may need to add export OMP_NUM_THREADS=1 to your shell environment (~/.bashrc), and/or in your job scripts, in order to avoid parallelization overload.

Example with Jupyter-notebook

Note

This should be used on a visualization server, see Using X2Go for data visualization.

Create and install a virtual environment dedicated to jupyter-notebook:

mkdir -p ~/tests/py39-jnb && cd ~/tests/py39-jnb
python3.9 -m venv ~/tests/py39-jnb
source bin/activate
python3 -m pip install jupyter
jupyter-notebook

Run it:

cd ~/tests/py39-jnb
source bin/activate
jupyter-notebook

Install python in your home

You can build your own python binary in $HOME, or team storage (see Where to store files?).

Download, from python.org, the version you need. Follow the installation instructions, using --prefix and altinstall.

Warning

The python3 from the system will always be available, so take care of $PATH and $PYTHONPATH to prioritize your own.

Note

Do NOT use --enable-optimizations if you intend to use this build on all PSMN clusters/partitions.

cd Python-3.9.4/
./configure --prefix=$HOME/py394
# [...]
make -j6
# [...]
make altinstall

Modify your $PATH and $PYTHONPATH environment variables as asked (in ~/.bashrc, export PATH="$HOME/py394/bin:$PATH", and do not forget to source it, or disconnect/reconnect).

To use ATLAS (Automatically Tuned Linear Algebra Software) library, add this to your environment (~/.bashrc):

export LD_LIBRARY_PATH="/usr/lib/atlas-base:$LD_LIBRARY_PATH"

then you can install python with ATLAS support.

You can now use this python binary, preferably in a virtual environment (see above).