Python¶
On the CÉCI clusters, the Python interpreter is installed, like the other software tools, with EasyBuild. EasyBuild allows installing multiple versions of Python, which can be loaded through “modules” with the ml <module name> command. The list of Python modules can be obtained with ml spider Python:
[dfr@lemaitre3 ~]$ ml spider Python
-------------------------------------------------------------------------------------------------------
Python:
-------------------------------------------------------------------------------------------------------
Description:
Python is a programming language that lets you work more quickly and integrate your systems more
effectively.
Versions:
Python/2.7.11-intel-2016a
Python/2.7.12-foss-2016b
[...]
Python/3.9.6-GCCcore-11.2.0-bare
Python/3.9.6-GCCcore-11.2.0
-------------------------------------------------------------------------------------------------------
To find other possible module matches execute:
$ module -r spider '.*Python.*'
-------------------------------------------------------------------------------------------------------
For detailed information about a specific "Python" package (including how to load the modules) use the mo dule's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:
$ module spider Python/3.9.6-GCCcore-11.2.0
-------------------------------------------------------------------------------------------------------
Having chosen a specific module, you will need to find out in which release it is available, with the more specific ml spider <full module name> command:
[dfr@lemaitre3 ~]$ ml spider Python/3.9.6-GCCcore-11.2.0
-------------------------------------------------------------------------------------------------------
Python: Python/3.9.6-GCCcore-11.2.0
-------------------------------------------------------------------------------------------------------
Description:
Python is a programming language that lets you work more quickly and integrate your systems more
effectively.
You will need to load all module(s) on any one of the lines below before the "Python/3.9.6-GCCcore-11.2.0" module is available to load.
releases/2021b
[...]
Warning
Make sure to understand the module systems and the concept of releases before reading on. They are explained in the Using pre-installed software section.
Note
When looking you are loading and unloading modules to test them, you can use the ml purge command to reinitialise the environment and remove all loaded modules.
Scientific Python and the SciPy-bundle¶
Python is a generic programming language ; to be useful for scientific computing, its library must be extended through packages. The most used and known Python packages that are necessary for scientific computing are numpy and scipy.
Once a Python module is loaded, the packages that are installed in that specific Python module can be used. They can be listed with the pip freeze command:
[dfr@lemaitre3 ~]$ pip freeze
alabaster @ file:///usr/local[...]/alabaster/alabaster-0.7.12
appdirs @ file:///usr/local[...]/appdirs/appdirs-1.4.4
asn1crypto @ file:///usr/local[...]/asn1crypto/asn1crypto-1.4.0
atomicwrites @ file:///usr/local[...]/atomicwrites/atomicwrites-1.4.0
attrs @ file:///usr/local[...]/attrs/attrs-21.2.0
Babel @ file:///usr/local[...]/Babel/Babel-2.9.1
bcrypt @ file:///usr/local[...]/bcrypt/bcrypt-3.2.0
bitstring @ file:///usr/local[...]/bitstring/bitstring-3.1.7
blist @ file:///usr/local[...]/blist/blist-1.3.6
CacheControl @ file:///usr/local[...]/CacheControl/CacheControl-0.12.6
[...]
Note
In the older Python modules, Numpy and SciPy are part of the Python module so loading the Pyton module automatically enables them. On newer Python modules, this is not the case anymore.
In the recent EasyBuild releases, the Python module does not contain scientific packages. Rather, a specific extension module is available, named SciPy-bundle, which can be loaded directly to have a Python environment with Numpy, Scipy, Pandas, etc. That module will automatically load the Python module on which it was built.
[dfr@lemaitre3 ~]$ ml spider SciPy-bundle
-------------------------------------------------------------------------------------------------------
SciPy-bundle:
-------------------------------------------------------------------------------------------------------
Description:
Bundle of Python packages for scientific software
Versions:
SciPy-bundle/2019.03-foss-2019a
SciPy-bundle/2019.10-foss-2019b-Python-2.7.16
SciPy-bundle/2019.10-foss-2019b-Python-3.7.4
[...]
SciPy-bundle/2021.05-foss-2021a
SciPy-bundle/2021.10-foss-2021b-Python-2.7.18
SciPy-bundle/2021.10-foss-2021b
SciPy-bundle/2021.10-intel-2021b
-------------------------------------------------------------------------------------------------------
For detailed information about a specific "SciPy-bundle" package (including how to load the modules) use
the module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:
$ module spider SciPy-bundle/2021.10-intel-2021b
-------------------------------------------------------------------------------------------------------
The Python packages that the SciPy-bundle module offers can be listed with the ml help SciPy-bundle command:
[dfr@lemaitre3 ~]$ ml help SciPy-bundle
----------------------- Module Specific Help for "SciPy-bundle/2021.05-foss-2021a" ------------------------
Description
===========
Bundle of Python packages for scientific software
More information
================
- Homepage: https://python.org/
Included extensions
===================
Bottleneck-1.3.2, deap-1.3.1, mpi4py-3.0.3, mpmath-1.2.1, numexpr-2.7.3,
numpy-1.20.3, pandas-1.2.4, scipy-1.6.3
and can be verified of course with the pip command.
[dfr@lemaitre3 ~]$ pip freeze | grep SciPybundle
Bottleneck @ file:///usr/local[...]/foss-2021a/Bottleneck/Bottleneck-1.3.2
deap @ file:///usr/local[...]/foss-2021a/deap/deap-1.3.1
mpi4py @ file:///usr/local[...]/foss-2021a/mpi4py/mpi4py-3.0.3
mpmath @ file:///usr/local[...]/foss-2021a/mpmath/mpmath-1.2.1
numexpr @ file:///usr/local[...]/foss-2021a/numexpr/numexpr-2.7.3
numpy @ file:///usr/local[...]/foss-2021a/numpy/numpy-1.20.3
pandas @ file:///usr/local[...]/foss-2021a/pandas/pandas-1.2.4
scipy @ file:///usr/local[...]/foss-2021a/scipy/scipy-1.6.3
Note
Some specific Python packages can have their own extension module
Extension-only modules¶
Additionally, some specific Python packages such as for instance TensorFlow, or Matplotlib are made available as extension modules, which allows even to have multiple versions of TensorFlow available for the same version of Python.
[dfr@lemaitre3 ~]$ ml spider Tensorflow
-------------------------------------------------------------------------------------------------------
TensorFlow:
-------------------------------------------------------------------------------------------------------
Description:
An open-source software library for Machine Intelligence
Versions:
TensorFlow/1.12.0-foss-2018b-Python-3.6.6
TensorFlow/1.15.2-foss-2019b-Python-3.7.4
TensorFlow/2.2.0-foss-2019b-Python-3.7.4
TensorFlow/2.4.1-foss-2020b
-------------------------------------------------------------------------------------------------------
For detailed information about a specific "TensorFlow" package (including how to load the modules) use th
e module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:
$ module spider TensorFlow/2.4.1-foss-2020b
-------------------------------------------------------------------------------------------------------
Such extension modules include the packages on which the main package depends:
[dfr@lemaitre3 ~]$ ml help TensorFlow
------------------------- Module Specific Help for "TensorFlow/2.4.1-foss-2020b" --------------------------
Description
===========
An open-source software library for Machine Intelligence
More information
================
- Homepage: https://www.tensorflow.org/
Included extensions
===================
absl-py-0.10.0, astunparse-1.6.3, cachetools-4.2.0, dill-0.3.3, gast-0.3.3,
google-auth-1.24.0, google-auth-oauthlib-0.4.2, google-pasta-0.2.0,
grpcio-1.32.0, gviz-api-1.9.0, h5py-2.10.0, Keras-Preprocessing-1.1.2,
Markdown-3.3.3, oauthlib-3.1.0, opt-einsum-3.3.0, portpicker-1.3.1,
pyasn1-modules-0.2.8, requests-oauthlib-1.3.0, rsa-4.7, tblib-1.7.0,
tensorboard-2.4.1, tensorboard-plugin-profile-2.4.0, tensorboard-plugin-
wit-1.8.0, TensorFlow-2.4.1, tensorflow-estimator-2.4.0, termcolor-1.1.0,
Werkzeug-1.0.1, wrapt-1.12.1
It will furthermore load the modules that are necessary, among which, the``Python`` and SciPy-bundle modules.
User-installed packages¶
If you cannot find the package you need in the ones provided by the modules, you can install them by yourself as explained in the Installing languages extensions section.