Navigation

index

next |

previous |

CÉCI »

Creating an account

Note

In order to create an account, you must be connected, either directly (Wifi, cable), or indirectly (via SSH or VPN) to your university network.

If you are member of a university part of CÉCI :

The first step is to visit the Login Management [https://login.ceci-hpc.be] site and enter your email address. From then on, your email address will be associated uniquely with your CÉCI login. You will then shortly receive an email with a web link. Click on that link; you are brought to a web form where you need to enter your identification information. Your information is then reviewed by a system administrator from your university. Once your details are confirmed, your account will be created. As soon as it is created, you receive your private key to the clusters by email (an attached file named ‘id_rsa.ceci’).

See also: Create a CÉCI account [https://www.ceci-hpc.be/assets/pdf/renew-create-account.pdf] step by step tutorial.

If you want to invite a guest :

You can create an account for your collaborators who do not have an email from one of the five universities members of the CÉCI. You must have a permanent position within your university and be a registered CÉCI user. If you are not, please register yourself first as in the previous paragraph. By using this functionality, you bear the responsibility of the account created for your guest. Go to login.ceci-hpc.be/#invite_guest [https://login.ceci-hpc.be/#invite_guest], and enter your email address and the email address of your guest (we need the official institutional email address of the guest. Addresses ending in @gmail.com, for instance, or other generic email addresses are not allowed). You will have to fill in the form sent to your email address, and your guest will have to send a SSH public key to the system administrator of your university. Detailed instructions will be sent by email to your guest directly.

Warning

By requesting access to the CÉCI clusters, you accept the rules related to the Fair use of the CÉCI clusters [https://www.ceci-hpc.be/fair_use.html] .

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Connecting to the clusters

How to configure your laptop to ease access to the clusters.

If you have a Linux computer...

If you have a Mac...

If you have a Windows computer...

Connecting from UNIX-like environment (Linux, Mac OS X, Cygwin/MingW)

Clusters

Gateways

Procedure

Troubleshooting

Frequently encountered problems

Connecting from Windows environment

Setup MobaXTerm (Preferred)

Setup Windows Subsystem for Linux (Advanced)

Choosing and using a cluster wisely

The choice of the cluster

The choice of the nodes

The job parametrization

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Connecting to the clusters »

How to configure your laptop to ease access to the clusters.

Besides installing one or two programs, there is no reason why you would not be able to connect and use the clusters from your laptop or desktop computer. But, as the clusters are running Linux, things go smoother if your laptop is also running Linux.

Two software are needed to connect to the clusters: a terminal [https://en.wikipedia.org/wiki/Computer_terminal#Text_terminals] and an SSH [https://en.wikipedia.org/wiki/Secure_Shell] client.

The terminal is the piece of software that will allow you typing commands and reviewing the results, and the SSH client is the software that will allow you actually connect to the clusters.

A third piece of software is necessary for graphical applications that run on the clusters: an X server [https://en.wikipedia.org/wiki/X_Window_System].

If you have a Linux computer...

... then things are made easier as you can set up your laptop just like the clusters are configured and hence have an homogeneous experience running your experiments and analysing your results.

Linux comes with all you need to connect to the cluster with your SSH Key. Just get accustomed to the notion of Agent Forwarding [http://www.unixwiz.net/techtips/ssh-agent-forwarding.html] and Key Management software such as Key Ring [http://en.wikipedia.org/wiki/GNOME_Keyring] and KWallet [http://en.wikipedia.org/wiki/KWallet] and you will never have to type in a passphrase again, even to copy files from one cluster to another. If you prefer a command-line solution, Keychain [http://www.funtoo.org/Keychain] is a good alternative.

If you have customized your Bash environment, you can simply copy your .bashrc and .bash_profile (or the equivalent) file to the cluster and feel like home out there.

Copying data is done easily with the rsync [http://en.wikipedia.org/wiki/Rsync] command, and managing your code is done effectively with a tool such as Mercurial [http://en.wikipedia.org/wiki/Mercurial] or Git [http://en.wikipedia.org/wiki/Git].

One major difference, though, is that, on your laptop, you will most probably install software using packages (typically rpms or debs). On the cluster, you will not be able to do that. You will need to compile the software from sources. So it can be wise to take the habit of recompiling all the software you use on your laptop so that you have sufficient experience doing it on the cluster. Compiling several versions of the gcc compiler and getting them to co-exist peacefully is a very good exercise. It furthermore allows you to install the same version as is installed on the cluster so your compilation tests are consistent.

Note that most clusters run REHL [http://www.redhat.com/products/enterprise-linux/], or a derivative such as CentOS [http://www.centos.org/] or Scientific Linux [https://www.scientificlinux.org/], or Debian [http://www.debian.org/]. Those are preferred over more up-to-date-but-less-stable distributions as Ubuntu or Fedora.

If you have a Mac...

... then you are nearly as comfortable as if you had a Linux, except that standard libraries are different, and differently organized so compiling will be a different experience. Still, you can install Linux in a Virtual Machine, or setup a dual boot [https://help.ubuntu.com/community/MactelSupportTeam/AppleIntelInstallation].

If you want to run graphical applications the official X server for MacOS is named XQuartz [https://www.xquartz.org/]. You can install it by downloading the DMG installer from the website or by using homebrew [http://brew.sh].

If you have a Windows computer...

… then you will need to install some software no matter what.

We cover two possibilites to work on a Windows environment:

Install a UNIX like environment like MobaXterm, Cygwin or Mingw. You can follow our guide to Download and setup MobaXterm.

Follow the guide to setup the Windows Subsystem for Linux feature and install an available Linux distro (only Windows 10 after October 2017 build support this).

We strongly recommend following the MobaXterm option as that will provide a complete environment for connecting to the clusters and copying/retrieving files from them.

Of course, you still have the possibility to setup a dual boot system with Windows and Linux. Every Linux distribution will allow you to do so effortlessly.

A final option, discussed here, is to install a Linux virtual machine, for instance with software such as Virtual Box [https://www.virtualbox.org/]. This option is very interesting if you do not have administrator privileges in your computer. Just ask to have that software installed, and, within it, you will have no restriction in your Linux virtual machine. It furthermore allows you to save the state of your machine, which is interesting for full system backups and migration to another computer. The downside is that you will need a fairly strong hardware configuration, especially if you plan on using several virtual machines in parallel (for instance to debug MPI programs).

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Connecting to the clusters »

Connecting from UNIX-like environment (Linux, Mac OS X, Cygwin/MingW)

This short tutorial describes how to connect to a CÉCI cluster from a terminal application, such as XTerm, Gnome Terminal, Konsole, iTerm, Linux running on Windows Subsystem For Linux, the native SSH client in Powershell (Windows 10 1809 and higher) etc.

Clusters

All CÉCI users can connect to any of the following clusters:

Lemaitre3 (UCL): lemaitre3.cism.ucl.ac.be

NIC5 (ULiège) : nic5.uliege.be

Hercules2 (UNamur): hercules.ptci.unamur.be

Dragon1 (UMons): dragon1.umons.ac.be

Dragon2 (UMons): dragon2.umons.ac.be

To decide which cluster is better suited for your kind of jobs, please check the details about them in the CÉCI clusters page [https://www.ceci-hpc.be/clusters.html].

Gateways

As the clusters are not accessible from outside the university networks, you will need to use a SSH gateway and the university VPN in some cases. Going through a SSH gateway can be entirely transparent provided your client is correctly configured. The CÉCI does not offer a centralized gateway service, so you will have to use the one provided by the University you belong to:

UCL: use gwceci.cism.ucl.ac.be as SSH gateway with your CÉCI account. See the CISM documentation [http://www.cism.ucl.ac.be/doc/_contents/Other/index.html#ssh-gateways].

ULB: use gwceci.ulb.ac.be as SSH gateway with your CÉCI account. You will need to use a VPN also if you are outside Belgium.

UMons: contact the local support team to learn how to setup the VPN, then use dragon2.umons.ac.be as gateway.

UNamur: use gwceci.unamur.be with your UNamur identity (eID)

ULiège: use gwceci.uliege.be with your CÉCI account AND use the ULiège VPN [https://my.segi.uliege.be/cms/c_11650735/fr/mysegi-new-vpn].

Choose your favorite terminal application to follow the steps below.

Procedure

1. Get the private CÉCI key

To get the private key, you need to create or renew your CÉCI account

Note

Please note that since the moment you got the email with you key confirming your account is active, you might need to wait up to ~20 minutes maximum until the access is enabled in all the clusters.

For some Universities the ceci public key needs to be set in the gateway and this can take some hours.

Once you have received your private key by email, store it in a safe location. The most rational place to store it is in your .ssh folder in your home directory, along with other SSH keys you may have. In case the file was downloaded in the Downloads folder in your home, move it with the command:

mkdir -p ~/.ssh && mv ~/Downloads/id_rsa.ceci ~/.ssh

Change the permissions of the file so that only you can read it:

chmod 600 ~/.ssh/id_rsa.ceci

If you don’t do this your SSH client will refuse to use the key giving an error which mentions Permisions 0644 for... are too open.

Create your public key:

ssh-keygen -yf ~/.ssh/id_rsa.ceci > ~/.ssh/id_rsa.ceci.pub

You will be asked by the passphrase you chose on the CÉCI account creation form.

Warning

If for the last step you get an error mentioning either Load key "id_rsa.ceci": invalid format or Load key "id_rsa.ceci": error in libcrypto, it should be fixed by running the command:

dos2unix ~/.ssh/id_rsa.ceci

see here for more information.

2. Connect to a cluster

Even if you are used to use ssh on the command line, the fact of having to go through a gateway makes the command to connect too cumbersome to type it every time. So we will instruct you on how to create a ssh_config file to ease with this.

Go to our ssh_config wizard that will aid in creating the configuration depending on your University.

Choose which is your University and provide the required information.

Use your favorite text editor on the terminal (nano, vim, emacs ...) to create or edit the configuration file ~/.ssh/config e.g.:

nano ~/.ssh/config

and copy-paste the contents provided by the wizard. If you already have contents there, keep it and paste this block below it, save the changes (ctrl+x then answer yes by pressing y and after that enter). Make sure the file has correct permissions:

chmod 600 ~/.ssh/config

Now everything is set, you can do your first connection using the command:

ssh cecicluster

Where you should replace cecicluster by one of the Host alias for the CÉCI clusters: lemaitre3, hercules, dragon1, dragon2 or nic5.

Note

If it is the first time you use the private key and/or your agent is not running you will be asked for the key passphrase twice. One for the gateway connexion and another for the cluster connexion.

If everything worked properly, you should be greeted by a message similar to

Welcome to

 ██╗     ███████╗███╗   ███╗ █████╗ ██╗████████╗██████╗ ███████╗██████╗
 ██║     ██╔════╝████╗ ████║██╔══██╗██║╚══██╔══╝██╔══██╗██╔════╝╚════██╗
 ██║     █████╗  ██╔████╔██║███████║██║   ██║   ██████╔╝█████╗   █████╔╝
 ██║     ██╔══╝  ██║╚██╔╝██║██╔══██║██║   ██║   ██╔══██╗██╔══╝   ╚═══██╗
 ███████╗███████╗██║ ╚═╝ ██║██║  ██║██║   ██║   ██║  ██║███████╗██████╔╝
 ╚══════╝╚══════╝╚═╝     ╚═╝╚═╝  ╚═╝╚═╝   ╚═╝   ╚═╝  ╚═╝╚══════╝╚═════╝

              Massively parallel CISM-CECI cluster

      80 nodes: 2 x 12-core Intel Skylake 5118@2.3GHz, 96GB RAM
            1:3-blocking OmniPath Architecture network

           contact, support: egs-cism@listes.uclouvain.be
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CÉCI clusters: Hmem - Lemaitre3 - Dragon1 - Dragon2 - Hercules - Vega - NIC4

      You currently have 0 job running, 0 pending.

      Don't know where to start?
               --> https://www.ceci-hpc.be/install_software.html
               --> https://www.ceci-hpc.be/slurm_tutorial.html

Note

Upon your very first connection to each CÉCI cluster, you will be greeted by a warning such as:

The authenticity of host 'my-ceci-cluster.example.com (XXX.XXX.XX.XX)' can't be established.
RSA key fingerprint is SHA256:aqUPC4C8gkBjgCUMpHt3kzpciSkQcKR2gNOahnbZN3c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ceci-cluster.ac.be,XXX.XXX.XX.XX' (RSA) to the list of known hosts.

This warning is normal, the SSH program warns that it is the first time it sees this computer. To make sure you are actually connecting to the right machine, you should compare the RSA key fingerprint [https://en.wikipedia.org/wiki/Public_key_fingerprint] shown in the message with the fingerprint announced on the Access section of the CÉCI cluster page [https://www.ceci-hpc.be/clusters.html]. The fingerprint can be based on hash function MD5 or SHA-256.

If they match, you are safe to proceed and enter yes. That hostname key will be stored and used to check in every subsquent SSH connection that the server is indeed always the same.

Note

If you hit an error that mentions something as:

line 12: Bad configuration option: proxyjump

Then change the line with the ProxyJump option in the ~/ssh/.config to read instead:

ProxyCommand ssh gwceci -W %h:%p

3. Using an SSH agent to avoid typing the passphrase at each login

You can further ease the process by using an SSH agent which will remember the passphrase so you do not have to type it each time you issue the SSH command. In any modern Mac OS or Linux distribution is quite certain that you have an agent already running, if the command:

ssh-add -l

returns The agent has no identities or a line like

2048 SHA256:1S30t9SSWSd2PIF0QVAPj1Xhtnj2UYqPw32ZIAu0MkI ~/.ssh/id_rsa.ceci (RSA)

you don’t have anything else to do, your agent is running. Please, check the notes at the bottom of this section if you are still being requested the passphrase at each cluster login.

If the command responds Could not open a connection to your authentication agent, you can start an agent with:

eval $(ssh-agent)

Then, load the CÉCI key with the command:

ssh-add ~/.ssh/id_rsa.ceci

you’ll be prompted once for your passphrase. From now on, in the current terminal, all ssh commands will be handled by the agent and you won’t have to type your passphrase again.

With the agent running, you can use the -A option of ssh to forward your agent from one computer to another (you don’t need this if you are using the ForwardAgent option in a .ssh/config as explained in the section above). This allows you to connect, or copy files, from one cluster to another effortlessly.

You can have an ssh-agent started automatically at login by using password managing software such as Mac OS Keychain [https://en.wikipedia.org/wiki/Keychain_%28Mac_OS%29], KDE KWallet [https://en.wikipedia.org/wiki/KWallet], Gnome Keyring [https://en.wikipedia.org/wiki/GNOME_Keyring], Seahorse [https://en.wikipedia.org/wiki/Seahorse_(software)], etc.

Note

In Mac OS, after Sierra update 10.12.2, in order to automatically unlock your keys these extra options are required on top of your ~/.ssh/config file:

Host *
    AddKeysToAgent yes
    UseKeychain yes

Troubleshooting

Step 1. Make sure that your key file has the correct permissions:

ls -l ~/.ssh/id_rsa.ceci

should output something that starts with

-rw------

Step 2. Make sure your key and your passphrase match by issuing the following command:

ssh-keygen -yf ~/.ssh/id_rsa.ceci

If the output is ‘load failed’ you either have a wrong passphrase or your key file has been corrupted.

Step 4. Make sure the ssh client is asking for a passphrase and not a password. Otherwise, it means you are not correctly instructing your ssh client to use the private key.

Run the ssh command with the -v option. You should find a line stating

debug1: Offering RSA public key: .ssh/id_rsa.ceci

The line immediately following that one should be something like:

debug1: Server accepts key: pkalg ssh-rsa blen 277

otherwise it means the key your are using does not correspond to the one stored on the cluster. If this happens while you just updated or renewed your account, it may indicate that your new key is not fully propagated yet. Try again later.

Frequently encountered problems

Here are some frequent errors you should avoid.

The permissions on your key file are not correct

If, after running ssh cecicluster, for instance, you see something like:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/home/ceciuser/.ssh/id_rsa.ceci' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: /home/ceciuser/.ssh/id_rsa.ceci
ceciuser@ceci-cluster.ac.be's password:

it means that Permissions 0644 for the /home/ceciuser/.ssh/id_rsa.ceci file are too open. Change them to 600 as explained in the first section of this document.

You did not specify the correct path to your SSH private key

If, after running ssh, you are being asked for a password directly,

$ ssh cecicluster
ceciuser@ceci-cluster.ac.be's password:

it means that your SSH client did not try to use the SSH key. Make sure you either used the -i option or that your ~/.ssh/config file is properly configured and contains no typos.

You used a wrong username or tried to connect before your keys are synchronised

If, after running ssh, you are being asked for a passphrase, then a password,

$ ssh cecicluster
Enter passphrase for key '/home/ceciuser/.ssh/id_rsa.ceci':
ceciuser@ceci-cluster.ac.be's password:

it often means that the user name you are using is not the correct one.

If you just requested or renewed your account, it could also mean that you are using the new private key while it has not been synchronized to that cluster yet (keep in mind that the clusters are not synchronized simultaneously). Wait about 15 minutes and try again.

You have too many SSH keys lying around

If the server responds with

$ ssh cecicluster
Received disconnect from host: 2: Too many authentication failures for ceciuser

that means that your client offered the server too many keys to try and access it. To make sure the client uses only the correct key (i.e. the one specified by the IdentityFile option,) you will need to add IdentitiesOnly yes at the top of your ~/.ssh/config file, or pass it as an argument:

$ ssh -o 'IdentitiesOnly yes' cecicluster

Note that for the SSH-agent to continue working properly when this option is added, you will need the public SSH key to be present in the .ssh directory.

You can generate the public SSH key from the private key (the one you received from the CÉCI with the following command:

ssh-keygen -yf ~/.ssh/id_rsa.ceci > ~/.ssh/id_rsa.ceci.pub

You copy/pasted the config file to a word processor

If the SSH client complains that

Bad configuration option: \302\240\302\240\302\240\302\240hostname

you should fix your ~/.ssh/config file.

The spaces in your ~/.ssh/config file are so-called “non breaking spaces” [https://unix.stackexchange.com/questions/272161/octals-302-240-together-seem-to-correspond-to-non-breaking-space] probably inserted there by a word processor.

You can try removing and re-inserting the spaces in the configuration file using a regular text editor or use GNU sed

sed -i .bak 's/\xc2\xa0/ /g' ~/.ssh/config

Make sure to use GNU sed, this will not work with the BSD sed that ships with MacOS. You can use Perl as an alternative:

perl -i.bak -pe 's/\xc2\xa0//g' ~/.ssh/config

Both options will create a backup ~/.ssh/config.bak

The passphrase in GNOME keyring was not properly updated

If the SSH client complains that

sign_and_send_pubkey: signing failed: agent refused operation

it could be that the passphrase stored in the GNOME keyring is not correct. You can re-enter it by first flushing them :

gnome-keyring-daemon -r -d

and then re-enter them:

ssh-askpass

The SSH private key was tampered with by Outlook

If ssh-keygen complains that

Load key "id_rsa.ceci": invalid format

or (more recent versions of OpenSSH)

Load key "id_rsa.ceci": error in libcrypto

It could be because the file was “translated” automatically, when you downloaded it to your computer, to the DOS text file type that uses a different convention for line breaks [https://en.wikipedia.org/wiki/Newline] than UNIX/Linux text files.

You can check that is the case with

cat -e id_rsa.ceci | grep -c "\^M"

If the output is non-zero, your file is indeed in the wrong format. The expected output should be 0. To fix it, use the dos2unix command:

dos2unix id_rsa.ceci

If dos2unix is not installed and you cannot or do not want to install it, check in any of the following command is installed: tr, col, or sed. If any of those command is installed:

Step 1: run the following command

cp id_rsa.ceci id_rsa.ceci.bak

before to make a copy of the downloaded file; and then

Step 2: run either of

col -b < id_rsa.ceci.bak > id_rsa.ceci

or

tr -d '\r' < id_rsa.ceci.bak > id_rsa.ceci

or

sed 's/\r$//' id_rsa.ceci.bak > id_rsa.ceci

depending on what you have available.

Alternatively, if vim is installed, you can also get the same result by opening the file with vim, then typing :set ff=unix; and then saving the file.

The resulting id_rsa.ceci file should then pass the cat -e id_rsa.ceci | grep -c "\^M" test and be accepted by the SSH client.

The MobaXterm version is too old

If MobaXterm complains that it

Couldn't load private key (unrecognised cipher name)

it is probably because the version you have is too old and does recognise the new SSH ciphers [https://en.wikipedia.org/wiki/Cipher] that are introduced in the more recent OpenSSH version used since the upgrade of the CÉCI account management system.

Try upgrading MobaXterm to version 23.2 or more recent.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Connecting to the clusters »

Connecting from Windows environment

We cover two different options to work from a Windows environment.

We recommend following the MobaXTerm option as that will provide a complete environment for connecting to the clusters, to copy/retrieve files from them and to launch graphical apps requiring an X server:

Setup MobaXTerm (Preferred)

If you are an experienced user of the Linux command line, you might find interesting to try the recent feature Windows Subsystem for Linux, note that only Windows 10 after October 2017 build support this:

Setup Windows Subsystem for Linux (Advanced)

Note

Please note that since the moment you got the email confirming your account is active, you might need to wait up to ~20 minutes maximum until the access is enabled in all the clusters.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Connecting to the clusters »

Connecting from Windows environment »

Download and setup MobaXterm

Go to the MobaXterm website [https://mobaxterm.mobatek.net] and download the free version. Make sure to select the prtable edition from the download page. Create a MobaXterm folder on your Windows user Documents one and uncompress inside it the contents of the downloaded zip file. We chose to create the MobaXterm folder on that particular location and we will refer to that on this guide. You can, of course, select some other location of your preference if you wish. The steps showed here has been done with version 20.2 of MobaXterm. Some behaviors may change between version.

Configuration information

Clusters

All CÉCI users can connect to any of the following clusters:

Lemaitre3 (UCL): lemaitre3.cism.ucl.ac.be

NIC5 (ULiège) : nic5.uliege.be

Hercules2 (UNamur): hercules.ptci.unamur.be

Dragon1 (UMons): dragon1.umons.ac.be

Dragon2 (UMons): dragon2.umons.ac.be

We will use my-ceci-cluster.example.com as a generic dummy address throughout the guide. When trying to follow the steps on your machine replace that by a real remote host of the CÉCI cluster you wish to use.

To decide which cluster is better suited for your kind of jobs, please check the details about them in the CÉCI clusters page [https://www.ceci-hpc.be/clusters.html].

Gateways

As the clusters are not accessible from outside the university networks, you will need to use a SSH gateway and the university VPN in some cases. Going through a SSH gateway can be entirely transparent provided your client is correctly configured. The CÉCI does not offer a centralized gateway service, so you will have to use the one provided by the University you belong to:

UCL: use gwceci.cism.ucl.ac.be as SSH gateway with your CÉCI account. See the CISM documentation [http://www.cism.ucl.ac.be/doc/_contents/Other/index.html#ssh-gateways].

ULB: use gwceci.ulb.ac.be as SSH gateway with your CÉCI account. You will need to use a VPN also if you are outside Belgium.

UMons: contact the local support team to learn how to setup the VPN, then use dragon2.umons.ac.be as gateway.

UNamur: use gwceci.unamur.be with your UNamur identity (eID)

ULiège: use gwceci.uliege.be with your CÉCI account AND use the ULiège VPN [https://my.segi.uliege.be/cms/c_11650735/fr/mysegi-new-vpn].

We will use my-univ-gateway.example.com as a generic dummy address throughout the guide. When following the steps on your machine replace it by the correct ssh gateway of your university of origin.

1. Get the private CÉCI key

To get the private key, you need to create or renew your CÉCI account

Note

Please note that since the moment you got the email with you key confirming your account is active, you might need to wait up to ~20 minutes maximum until the access is enabled in all the clusters.

For some Universities the ceci public key needs to be set in the gateway and this can take some hours.

Once you have received your private key by email, store it the Documents\MobaXterm folder. You should have this configuration.

2. Setup a remote session

Double click the MobaXterm_Personal_XX.X.exe executable file inside the Documents\MobaXterm folder. You should see the MobaXterm main window like this one:

Click on the Session icon session icon on the top left corner.

The Session settings configuration panel will open, select the SSH session type ssh icon and you should see a window like this:

In the Remote host field introduce the cluster remote address for your desired cluster

Lemaitre3 (UCL): lemaitre3.cism.ucl.ac.be

NIC5 (ULiège) : nic5.uliege.be

Hercules2 (UNamur): hercules.ptci.unamur.be

Dragon1 (UMons): dragon1.umons.ac.be

Dragon2 (UMons): dragon2.umons.ac.be

Tick the Specify username box and introduce your CÉCI username. Click the Advanced SSH settings tab and you will see these extra options:

Tick the Use private key box and click on the icon session icon on that field, a file browser will be opened, locate your id_rsa.ceci key file you received by e-mail. Take note that you should not change the location of the CÉCI key file. If you move it you will have to update its location in this setting section again, to allow MobaXterm to find it. Check that the SSH-browser type is set as SFTP protocol. The SSH settings tab should look like this at the end of the process:

Select Network settings tab.

For Mobaxterm version 2.5 or earlier, tick the Connect through SSH gateway box and for any cluster always use the ssh gateway from your university:

UCL: gwceci.cism.ucl.ac.be

ULB: gwceci.ulb.ac.be

UMons: dragon2.umons.ac.be

UNamur: gwceci.unamur.be

ULiège: gwceci.uliege.be

In User add the login name to connect to the gateway. Tick the Use private key box and click on the icon session icon on that field, and locate your id_rsa.ceci key file again.

For Mobaxterm version 20.6 or higher, click on “SSH gateway (jump host)”

Fill the Gateway host with the gateway from your university (see the list above). In Username add the login name to connect to the gateway. Tick the Use SSH key box and click on the icon session icon on that field, and locate your id_rsa.ceci key file again.

Warning

If you are working at UMons use dragon2.umons.ac.be as gateway for all CÉCI clusters except for dragon2 itself. For this cluster do not fill the gateway tab contents.

Press the OK button and you should be prompted for your passphrase. Enter here the passphrase you chose while filling the CÉCI web application

You should connect to the cluster and be greeted by a screen similar to this one:

On the left sidebar you will see on the sftp icon tab a file browser of your home directory in the cluster. You will see by default many files whose names start with a dot (.) symbol. These are hidden files of the Linux environment and you should not delete nor move them. You can edit and customize them if you are aware of what you are doing.

When you disconnect from the cluster (by typing exit or closing the terminal tab) you will find on the left sidebar on the Sessions tab a shortcut to the session you just setup. From now on, when you open MobaXterm, you can just double click that shortcut and you will start a remote session on my-ceci-cluster.example.com.

To create new sessions for other clusters:

Right click the shortcut on the Sessions tab and select Duplicate session.

Right click the shortcut of the new duplicated session and select Edit session

Change the cluster remote address in Remote host field (the others parameters do not change) and press the OK button.

Right click again in the shortcut and select Rename session to rename the session with the cluster name.

You can repeat step 9 for the rest of the CÉCI clusters you are planning to use.

If you wish you can also create a direct shortcut on your desktop. To do that, right click on the saved session name and choose Create a desktop shortcut, as shown below. An icon will appear on your Desktop which double clicking starts MobaXterm with a session on the cluster.

3. Copying files to and from the cluster

After you succeeded to setup the shortcut for connecting to a cluster, as we noted on the step 6 of the previous section, you will see on the left sidebar on the Sftp tab a file browser on the cluster you are connected to.

You can simply drag and drop files from your computer to that panel and they will be copied to the cluster. The same is valid for retrieving files from the cluster to your computer.

If you right click on that panel, you will see different options to interact with the browser (see the figure below). Remember always to press the Refresh current folder button after you copied something or a new file or folder is created on the cluster.

Note that you also have the right-click options directly available through the icons on the toolbar at the top of the panel.

4. Setup an SSH agent to avoid typing the passphrase at each login

After you successfully setup your cluster to connect by double clicking the shortcut, you will notice that you are prompted for the passphrase for each connection you make to a cluster. To avoid retyping it each time, you can setup an internal SSH agent in MobaXterm that will take care of unlocking the private key when you open the application.

Open the MobaXterm program and click to the Settings icon settings icon

You should see the MobaXterm Configuration panel. In the General tab choose the MobaXterm passwords management option, a new panel will be opened, make sure that Save sessions passwords has the options Always and Save SSH keys passphrases as well selected as shown below and click OK.

Now go in the same MobaXterm Configuration panel to the SSH tab. Make sure that all the options below the SSH agents section are ticked.

Press the + button in the Load following keys at MobAgent startup field and look for your id_rsa.ceci file, at the end the panel should like as this

Remember that you should not change the file location, otherwise you’ll have to come back to this settings to update it.

Press OK and you will be prompted for restarting MobaXterm, choose to do so.

When it is relaunched you will be asked for the private key passphrase at the beginning, after you introduced it correctly it will stay saved for all the forthcoming sessions. You should be able to double click in the shortcuts for the clusters and get connected directly.

Frequently encountered problems

You are using the wrong key

To ensure you are using the correct key we need to compare the finger print of your private key with the finger print of the public key on the CÈCI front-ends.

Open MobaXterm_Personal_XX.X.exe go to tools menu and select the ssh key generator icon ssh key generator. In Conversion tab, select Import key. and load your id_rsa.ceci key. You will be asked for the key passphrase.

The public key and the fingerprint will be displayed.

Key change after account renewal

After you have renewed your account, or requested a new key, you will have to replace the old key file with the new one you received by email. You can rename the old file to keep it rather than erasing it.

When that is done, you should be able to connect again using the configuration your previously set up.

There is one notable exception though, the SSH agent. You will have to replace the key in the agent with the new one.

Click ‘Settings’

Choose the SSH tab

Remove previous key by selecting it and hitting the - button, then click + and add the new file

Let MobaXTerm restart

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Connecting to the clusters »

Connecting from Windows environment »

Windows Subsystem for Linux

After the Windows 10 Fall Creators Update (FCU) released on October 17th 2017 Microsoft started to officially support on Windows OS the Windows Subsystem for Linux [https://docs.microsoft.com/en-us/windows/wsl/about] (WSL) feature. If you are running this Windows 10 build or later, after enabling this feature, you will be able to install one or more Linux distros directly from the Windows Store.

With this kind of environment in place you will be able to connect to the clusters as explained in the how to connect from a UNIX-like environment guide.

Installation

To verify which Windows 10 build you are running open the Settings panel and go to System > About, you should see a screen similar as this one:

If your build is 16215 or later you are good to go. See the following steps to enable the feature and install a supported Linux distro.

Open the Settings panel and search for the Turn Windows Features on or off dialog as shown below:

In this dialog box look for the Windows Subsystem for Linux option and enable it. Click OK and you’ll be prompted to reboot, this is required WSL will not work until you’ve rebooted!

After restarting, open the Microsoft Store and look for an available Linux distribution. Here are direct links to some store installers:

Ubuntu [https://www.microsoft.com/store/p/ubuntu/9nblggh4msv6]

OpenSUSE [https://www.microsoft.com/store/apps/9njvjts82tjx]

You’ll see a screen like the following, select Get to start the installation:

Once the installation has completed, select Launch. This will open a console window. Wait for the installation to complete then you will be prompted to create your Linux user account as shown below:

Create your UNIX username and password. This user account has no relationship to your Windows username and password and hence both can be different. Learn more about your UNIX account [https://docs.microsoft.com/en-us/windows/wsl/user-support].

After you finished the setup follow the section how to connect from a UNIX-like environment to connect to the clusters. Notice that all your Windows partitions will be automatically mounted at the /mnt path inside the Linux environment, e.g. C:\ will be at /mnt/c.

Set up X11 forwarding

As windows does not have an X11 server installed by default you’ll need to install a package providing one. You can use the VcXsrv software [https://sourceforge.net/projects/vcxsrv/] for that. We’ll assume in the rest of this guide that you are using the Ubuntu Linux app provided in the Windows Store.

Download the latest VcXsrv version from here [https://sourceforge.net/projects/vcxsrv/files/vcxsrv/], make sure to choose the vcxsrv-64.version.number.installer.exe file. Follow the install procedure and go for the full install.

Launch the program, you’ll get a wizard setup, leave all the default options, in the final screen choose Save configuration, you’ll be prompted to save a file that will work as a launcher with those options, is reasonable to save it on your Desktop.

Open the Ubuntu Linux app and install the following packages:

sudo apt install xauth x11-common x11-apps ssh

add export DISPLAY=localhost:0 to your .basrc file, you can use the following command:

echo "export DISPLAY=localhost:0" >> ~/.bashrc

Restart the terminal, this is an important step, the settings will not take effect until you close it and open a new one.

In the new terminal, you can verify that your local Xserver is working by running xcalc or xeyes. To connect to the clusters it is recommended to use an SSH config file [https://www.ceci-hpc.be/sshconfig.html] and make sure you have the following options active for them:

ForwardAgent yes
ForwardX11 yes
ForwardX11Trusted yes

Login in the cluster, you should be able to run your graphical application. If you get some warnings during the login about No xauth data you can add the line XauthLocation /usr/bin/xauth at the bottom of the /etc/ssh/ssh_config file in your local Ubuntu Linux environment.

Take note that in future use you should always have an active X server running (make sure to have the xserver icon in your taskbar) before running the Ubuntu Linux app.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Connecting to the clusters »

Choosing and using a cluster wisely

In order to ensure a fair access to the computing resources to everyone, it is important that resources be used at the maximum of their possibilities.

Indeed, not all users are scientifically born equal. Some need to solve problems that parallelise embarrassingly easily, while other must design complex parallel algorithms with a lot of communication, or need to write awful loads of data to disks. Some have accessibility to well-written software designed for parallel use on a cluster, or have the resources to design such a software, while others must rely on not-so-great software that unfortunately the best to solve their problem.

Therefore, it is important that resources which are crucial to a job are not monopolized by another job which does not require them.

Note

By connecting to a CÉCI cluster, you accept the rules related to the Fair use of the CÉCI clusters [https://www.ceci-hpc.be/fair_use.html] .

The choice of the cluster

The CÉCI offers quite a diversity of clusters, each tailored for one or two job types. Those job types are detailed in the Clusters [https://www.ceci-hpc.be/clusters.html] page.

The general rules can be expressed as follows: if a cluster

has a fast network (e.g. Infiniband) and your job does not make intense use of network communication (i.e. with MPI), please prefer another one;

has a fast scratch space (e.g. Lustre, Fraunhoffer, GPFS) and your job does not produce loads of data, please prefer another one;

has large memory on the nodes (more than 300GB) and your job does not need that much RAM, please prefer another one;

has queues with large allowed running time (more than 7 days) and your job is able to checkpoint itself, or will typically run quicker than that, please prefer another one.

Of courses these are general guidelines, and if you see a cluster being underused, and you have jobs waiting, feel free to submit them on that cluster. But the sysadmins may or may not let your job run if it is deemed unfit.

The choice of the nodes

The choice of the nodes and of the placement of the processes on the nodes is subject to the same logic.

If a node

has an accelerator (e.g. GPU, FPGA, Xeon Phi) and your job is not able to take advantage of it, please prefer another one;

has a brand new CPU of the latest generation and your program is not compiled to take advantage of the particular features of that CPU family, please prefer another one;

has a specific software licensed attached to it and your job does not need it, please prefer another one;

has an extra large local scratch filesystem (more than 2TB) and your job does not read nor produce large amounts of data, please prefer another one.

According to the same ideas, if your job is using MPI and the network is fast, it very rarely makes sense to choose specifically the process placement (e.g. asking for 4 processes on 4 nodes) rather than letting the job manager decide for you as it will (1) try its best to put all the processes on the smallest amount of nodes, and (2) it is topology aware and will try its best to put all processes on nodes that are no further away than one hop. It only makes sense if each process is using a local resource heavily (e.g. a local filesystem) in the hope that the other jobs on the machine do not.

The job parametrization

Once more, to ensure a fair use of the resources, make sure that you specify as tightly as possible the resources you need, for instance in terms of running time, or memory. As a general rule, do not let the default values and do not copy the script of a colleague and use it blindly. Engage in what you are doing, understand what you are doing, and do it properly.

Letting the default memory, if you need less than the default, leads to memory not being used, and to jobs waiting unnecessarily for memory to become available.

Letting the default maximum running time, if your job is expected to need less than that, leads to your job not being eligible for backfill, starting later than it could have, and ultimately delaying the jobs with lower priority possibly for no reason. Always observe the running time of your jobs and try to learn how it is affected by the parameters and/or data size of your job.

Asking for many cores should only be done once you have the certainty that your job scales, that is that its running time is nearly inversely proportional to the number of CPU used. Try with a low number, time your jobs, and grow step by step the number of CPUs. If your job saturates at 8 CPUs and you request more than that, they are wasted and unavailable for the other jobs.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Introduction to Linux

To use the CÉCI cluster you must know some basics about Linux. Among others, you will need to know how to

List files and directories.

Move inside directories.

Create, copy or rename (move) files and directories.

Use a text editor.

The learning of Linux is outside the scope of this documentation. There are plenty of Linux tutorials available on the web and writing our own would bring no added value.

Here is a list of tutorials selected by CÉCI sysadmins

UNIX Tutorial for Beginners [http://www.ee.surrey.ac.uk/Teaching/Unix]

Linux command line beginner guide video playlist [https://www.youtube.com/playlist?list=PLj-2elZxVPZ_E5cFhNrUa_BqWC8sfcH6q]

Beginner’s Guide to the Bash Terminal video [https://www.youtube.com/watch?v=oxuRxtrO2Ag]

Interactive tutorials in the form of games

bashcrawl [https://gitlab.com/slackermedia/bashcrawl]

To start playing do:

git clone https://gitlab.com/slackermedia/bashcrawl.git
cd bashcrawl/entrance
cat scroll

and follow instructions. Also read the README file [https://gitlab.com/slackermedia/bashcrawl/-/blob/master/README.md]

GameShell [https://github.com/phyver/GameShell]

To start playing do:

wget https://github.com/phyver/GameShell/releases/download/latest/gameshell.sh
bash gameshell.sh

and follow instructions. Also read the README file [https://github.com/phyver/GameShell/blob/master/README.md]

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Copying files

Copying from UNIX-like environment (Linux, Mac OS X, Cygwin/MingW)

Copying from Windows environment

Copying using Cyberduck graphical sftp client (MacOS)

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Copying files »

Copying from UNIX-like environment (Linux, Mac OS X, Cygwin/MingW)

Prerequisite

To use examples in this section your ssh environment must be configured and an agent must be running.

Copying files back and forth

With your agent running, copying files is easy with, for instance the scp command and the rsync command. The former works exactly like the cp command, except that it works across the network to copy files from one computer to another. A command like

scp -r mydir/ cecicluster:

will copy the directory mydir on your personal computer to your home directory on the cluster cecicluster. To retrieve a file in the mydir folder in the cluster home, to a local folder on your computer you must use

scp cecicluster:mydir/file1 local/folder/

If your login on the cluster is not the same as the one on your local computer, and you did not specify your login in your ssh config file, make sure to specify your login with something like the following

scp -r mydir/ mylogin@cecicluster:

In the case you are not using an ssh agent and your ~/.ssh/config is not configured, you will need to also add the -i ~/.ssh/id_rsa.ceci option to the command line.

You can also use the rsync command. The latter is more intelligent in the sense that it only copies what is needed to be copied in the case the target directory already exist and contains unchanged files. Click here [http://www.thegeekstuff.com/2010/09/rsync-command-examples/] for some examples.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Copying files »

Copying from Windows environment

We cover two options for copying files to/from the clusters:

If you use MobaXterm to connect to the clusters following our guide, see the copying files section there.

If you are using the more advanced WSL option to connect, then you can use scp as from a UNIX environment.

We strongly recommend following the MobaXterm approach.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Copying files »

Copying using Cyberduck graphical sftp client (MacOS)

Before you setup Cyberduck, you will need to have your ~/.ssh/config file properly configured as explained in the Connecting to a UNIX-like computer section. You can use the Wizard [https://www.ceci-hpc.be/sshconfig.html] for that.

Once you have installed Cyberduck following the instructions on the Cyberduc website [https://cyberduck.io/download/], launch it.

Click on the + button to setup a new server, change the protocol type from FTP ...

to SFTP:

and fill only the Server: box with the same name as in your SSH configuration file. The image below shows the setup as remote server the nic5 cluster. The other fields should be automatically filled in if Cyberduck finds your ~/.ssh/config file. Close the window by clicking on the red button in the upper left corner.

Enter the CECI key passphrase when Cyberduck asks for it (possibly twice if you go through a gateway)

Accept the SSH host public key fingerprint after verifying it from the CÉCI website [https://www.ceci-hpc.be/clusters.html]

You should then see your file on the remote cluster, and be able to drag and drop files or folders to and from the cluster.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Editing files

Sooner or later, you will need to modify a file you have already uploaded to the cluster. One obvious option is to modify the file locally on your laptop and then re-upload it with SCP or another tool. This of course lacks convenience...

Other options include using a text editor with remote editing capabilities such as SublimeText, TextMate, Notepad++, TextWrangler, Atom, etc., or using the remote editing features of FileZilla, Cyberduck, etc. or mounting the remote filesystem with SSH-FS.

All these approaches have pros and cons, but the quickest and most efficient is to learn how to use a terminal-based editor such as Vim, Emacs, or Nano, or mcedit, ne, slap, micro, pico, Joe, mped, etc. You most probably will find Vim, Emacs and Nano preinstalled on every Linux system you can access. So learning one of those three is a must.

If you want super editing powers (read features), try both Vim and Emacs ; they are so different that you’ll soon feel one of them is your preferred choice. Then learn it inside out.

If you want very quick time to market (read ease of use), go for Nano or Pico.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Submitting jobs with Slurm

Resource sharing on a high-performance cluster dedicated to scientific computing is organized by a piece of software called a resource manager or job scheduler. Indeed, users do not run software on those cluster like they would do on a workstation, rather, they must submit jobs, which are run unattended, by the job scheduler at the time, and on the resources, decided by its algorithm.

Slurm [https://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management] is a resource manager and job scheduler designed to do just that, and much more. It was originally created by people at the Livermore Computing Center [https://computing.llnl.gov/], and has grown into a full-fledge open-source software backed up by a large community, commercially supported by the original developers [https://www.schedmd.com], and installed in many of the Top500 [https://www.top500.org] supercomputers.

Anatomy of a cluster

A high-performance compute cluster (HPC cluster) is a made of a set of powerful computers, called nodes in this context, linked together through a fast network. Multiple format factor exist, but one common one is the “pizza box” shape, depicted in the image below showing multiple such servers stacked in a data center rack.

A cluster typically hosts multiple types of nodes:

management nodes: these computers run the management services like Slurm, but also the user identities database, the monitoring and reporting applications, etc. ;

login nodes: these are the computers users login into through SSH to install software and submit jobs ;

storage nodes: these computers host user files in possibly multiple filesystems ; and

compute nodes: these are the computers where jobs run and where the computations actually take place. They are often organised into partitions

Alongside these most common types, a cluster can also comprise

visualization nodes: computers dedicated to data visualization with specific hardware ;

I/O nodes: computers dedicated to data ingress and outgress to and from the cluster.

The typical usage therefore involves connecting to a login node, making sure data are stored on the right filesystems and software are available, and then submitting jobs. The jobs are dispatched to the compute nodes by the resource manager based on the adequation between the resources resquested by the job and those offered by the compute nodes.

Warning

On a cluster, you do not run the computations on the login node. Computations belong on the compute nodes, when, and where decided by the scheduler.

Inside a compute node, the computation is carried on by the processors. A processor looks like this:

A typical compute node has one, two, or four sockets on the motherboard, that each can host a processor. The processors are made of multiple cores, that can be though of as independent computing units, each one being in turn possibly “divided” into two hardware threads.

Hardware threads are a technology designed to hide the latencies of the memory and feed the compute units fast enough to keep them busy all the time. It can be activated or deactivated depending on the cluster. Depending on the workload, it can benefit or slightly harm the computations times.

Compute nodes also provide volatile working memory, or RAM, where data are stored when they are being processed or produced, before they are enventually saved to permanent storage (disks).

Compute nodes can also offer accelerators, such as GPUs, which are pieces of hardware where heavy computations can be offloaded. GPUs are often much more powerfull than regular processors, but do not have the same flexibility and require specific programming skills.

Gathering cluster information

Slurm offers the sinfo command to get an overview of the resources offered by the cluster. By default, sinfo lists the partitions that are available. A partition is a set of compute nodes (computers dedicated to... computing) grouped logically based on either physical properties of the hardware or job scheduling policies. Typical examples include partitions dedicated to debugging where only small and short jobs can be scheduled, or partitions dedicated to visualization with nodes equipped with specific graphic cards.

# sinfo
PARTITION AVAIL TIMELIMIT NODES STATE  NODELIST
batch     up     infinite     2 alloc  node[08-09]
batch     up     infinite     6 idle   node[10-16]
debug*    up        30:00     8 idle   node[01-07]

In the above example, we see two partitions, named batch and debug. The latter is the default partition as it is marked with an asterisk. All nodes of the debug partition are idle, while two of the batch partition are being used. The nodes in this example are named node01 to node16.

The sinfo command also lists the time limit (column TIMELIMIT) to which jobs are subject. On every cluster, jobs are limited to a maximum run time, to allow job rotation and let every user a chance to see their job being started. Generally, the larger the cluster, the smaller the maximum allowed time.

If the output of the sinfo command is organised differently from the above, it probably means default options are set through environment variables or wrappers are used at your site to provide more suitable defaults. Or you could have a different Slurm version.

The command sinfo can output the information in a node-oriented fashion, with the argument -N.

# sinfo -N -l
NODELIST        NODES PARTITION     STATE  CPUS  S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
node[01-02]         2    debug*      idle  32    2:8:2   3448    38536     16 Intel   (null)
node[03,05-07]      4    debug*      idle  32    2:8:2   3384    38536     16 Intel   (null)
node03              1    debug*      down  32    2:8:2   3394    38536     16 Intel   "Disk replacement"
node[08-09]         2     batch allocated  32    2:8:2    246    82306     16 AMD     (null)
node[10-16]         7     batch      idle  32    2:8:2    246    82306     16 AMD     (null)

With the -l argument, more information about the nodes is provided, among which the number of “CPUs” (CPUS), which is the number of processing units that the jobs can use. It should generally correspond to the number of sockets (S) times number of cores per socket (C) times number of hardware threads per core (T in the S:C:T column) but can be lower in the case some CPUs are reserved for system use.

The other columns report the volatile working memory (RAM – MEMORY), the size of the local temporary disk (also called local scratch space – TMP_DISK), and the node “weight” (an internal parameter specifying preferences in nodes for allocations when there are multiple possibilities). The last but one column (AVAIL_FE) show so-called features of the nodes, that are set by the administrator, and can refer to a processor vendor or family, a specific network equipment, or any desirable feature of the node, that can be used to choose one node type to another.

The last column, (REASON), if not null, describes the reason why a node would not be available.

Note

You can actually specify precisely what information you would like sinfo to output by using its --format argument. For more details, have a look at the command manpage with man sinfo.

Anatomy of a job

Jobs are made of one or multiple sequential steps, each consisting in one or multiple parallel tasks that will be dispatched to possibly distinct nodes.

Each task will be allocated CPUs, memory, and possible other generic resources in an exclusive manner by Slurm. Two jobs cannot share the same resource unless explicitly forced by the admins, but that is generally not the case. Therefore, jobs can only start when all needed resources are free and not needed by another higher priority job. Jobs are indeed assigned a priority when they are submitted, which can depend upon multiple factors. More on that later.

For the scheduling process to work properly, you will need to describe your job before you submit it:

what are the steps (i.e. which program must be run and how) ;

how many tasks there will be ;

what resource each task needs (CPU, memory, etc.), and

for how long the job is supposed to run.

All of these, along with potentially aditionnal job parameters submission options, can be described in a submission script.

A submission script is a shell script [https://en.wikipedia.org/wiki/Shell_script], e.g. a Bash [https://en.wikipedia.org/wiki/Bash_(Unix_shell)] script, whose comments, if they are prefixed with #SBATCH, are understood by Slurm as parameters describing resource requests and other submissions options. You can get the complete list of parameters from the sbatch manpage man sbatch.

Important

The #SBATCH directives must appear at the top of the submission file, before any other line except for the very first line which should be the shebang [https://en.wikipedia.org/wiki/Shebang_(Unix)] (e.g. #!/bin/bash).

The script itself is a job step. Other job steps are created with the srun command, that takes as argument the name and options of the program to run.

Note

If there is only one step, the srun command can be omitted, but it has consequences in how signals are handled and how accurate reporting is. It is often best to use it in all cases.

For instance, the following script, hypothetically named submit.sh,

#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=res.txt
#SBATCH --partition=debug
#
#SBATCH --time=10:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=100

srun hostname
srun sleep 60

describes a job made of 2+1 steps (the submission script itself plus two explicit steps created by calling srun twice), each step consisting of only one task that needs one CPU and 100MB of RAM. The first step will run the hostname command, and the second one the useless sleep command. The job is supposed to run for 10 minutes on the debug partition, be named test, and create an output file named res.txt.

Important

It is important to note that as the job will run unattended, it will not be attached to a terminal (screen) so everything that the program that is run would like to write to the terminal will be redirected by Slurm to a file, whose name is specified by the --output parameter. (Note that for the same reasons, the program cannot expect input from the user through the keyboard to run properly.)

Once the submission script is written properly, you need to submit it to slurm through the sbatch command, which, upon success, responds with the jobid attributed to the job. (The dollar sign below is the shell prompt [https://en.wikipedia.org/wiki/Unix_shell#Bourne_shell])

$ sbatch submit.sh
sbatch: Submitted batch job 12321

Warning

Make sure to submit the job with sbatch and not bash; also do not execute it directly. This would ignore all resource request and your job would run with minimal resources, or could possible run on the frontend rather than on a compute node.

The job then enters the queue in the PENDING state. You can verify this with

$ squeue --me

Once resources become available and the job has highest priority, an allocation is created for it and it goes to the RUNNING state. If the job completes correctly, it goes to the COMPLETED state, otherwise, it is set to the FAILED state.

Why is my job not starting?

A job can stay in the PENDING state for a long time, especially if the maximum allowed time on the cluster is large. It is always important though to check the reason for which the job is pending. If the reason is Priority, then there is not much you can do, but there are other possible reasons, some of which require action on your part.

The reason is listed in the last column of the default output of squeue and the description of the reason codes is available from the documentation [https://slurm.schedmd.com/squeue.html#SECTION_JOB-REASON-CODES].

Gathering job information

The squeue command shows the list of jobs which are currently running (they are in the RUNNING state, noted as ‘R’) or waiting for resources (noted as ‘PD’, short for PENDING).

# squeue
JOBID PARTITION NAME USER ST  TIME  NODES NODELIST(REASON)
12345     debug job1 dave  R   0:21     4 node[09-12]
12346     debug job2 dave PD   0:00     8 (Resources)
12348     debug job3 ed   PD   0:00     4 (Priority)

The above output shows that one job is running, whose name is job1 and whose jobid is 12345. The jobid is a unique identifier that is used by many Slurm commands when actions must be taken about one particular job. For instance, to cancel job job1, you would use scancel 12345. Time is the time the job has been running until now. Node is the number of nodes which are allocated to the job, while the Nodelist column lists the nodes which have been allocated for running jobs. For pending jobs, that column gives the reason why the job is pending. In the example, job 12346 is pending because requested resources (CPUs, or other) are not available in sufficient amounts, while job 12348 is waiting for job 12346, whose priority is higher, to run. Each job is indeed assigned a priority depending on several parameters whose details are explained in section Slurm priorities. Note that the priority for pending jobs can be obtained with the sprio command. Other reasons can exist ; they are listed in the Slurm documentation [https://slurm.schedmd.com/squeue.html#SECTION_JOB-STATE-CODES].

There are many switches you can use to filter the output by user --user, by partition --partition by state --state etc. As with the sinfo command, you can choose what you want sprio to output with the --format parameter.

Interestingly, you can get near-realtime information about your job when it is running (memory consumption, etc.) with the sstat command, by running sstat -j jobid. You can select what you want sstat to output with the --format parameter. Refer to the manpage for more information man sstat.

Upon completion, the output file contains the result of the commands run in the script file. In the above example, you can see it with cat res.txt command.

The example shown above illustrates a serial job which runs a single CPU on a single node. It does not take advantage of multi-processor nodes or the multiple compute nodes available with a cluster, by contrast to the example jobs in the squeue output that use multiple nodes.

The next sections explain how to create parallel jobs.

Going parallel

There are several ways a parallel job, one that leverages multiple compute units (CPUs) at the same time, can be created.

Warning

The type of job that can be created depends on the capabilities of the software being used. Parallel computing requires specific programming techniques, which, if not used, lead to simply the same computation being performed multiple times, with no actual benefit.

If the resource request fit with the capabilities of the program, then Slurm will be able to allocate every parallel sequence of computing instructions, called a thread in programming, to a different CPU in a precise one-to-one mapping. This process is called “binding”, or “pinning”. If there are more threads created by the program than there are CPUs available, the operating system will iteratively start and stop threads to allocate CPU time to every thread, in a procedure called “context switching”. That procedure consumes resource in itself and incurs significant overhead to be considered undesirable for high-performance computing.

Single-node parallelism

In the Linux operating system, running a program will initiate a process, that is a running copy of the program in the main memory. A process initially consists of one single thread, but it can clone itself into multiple threads that all share the same memory space (shared-memory programming) and can perform computations in parallel, but it can also spawn (“fork” in Linux terms) other processes, whose memory spaces are independent and whith which it can communicate internally.

Multithreaded programms can be written using the pthreads [https://en.wikipedia.org/wiki/Pthreads] system call, but most multithreaded scientific software is written with OpenMP [https://en.wikipedia.org/wiki/OpenMP]. OpenMP allows creating Single program, Multiple data programs, or SPMD. The same program is run multiple times, but each instance is identified by a thread ID, or rank that can be used to differenciate the behavior of each instance, for instance working on a separate subset of the data. Optimised linear algebrae of signal processing libraries use OpenMP a lot for parallelism.

Note

Even if the software you write is purely sequential and does not use OpenMP for instance, if it links to optimized scientific libraries (as is the case for interpreted languages like Python, R or Julia, they will use OpenMP behind the scenes so you should request at least 4 or 8 CPUs.

Multi-node parallelism

Multi-node parallelism assumes distributed memory programming with either

no communication between the processes on the different nodes ; or

message passing through the network ; or

communication through a common filesystem.

The first case, called embarrassingly parallel, simply relies on multiple instances of the same program being started with either - a different environment, and possibly - different arguments.

The second case is typical of high-performance computing, and requires specific large-bandwith low-latency, network hardware. Most message passing scientific software use MPI [https://en.wikipedia.org/wiki/Message_Passing_Interface], a library that takes care of instantiating multiple instances of the same program on different nodes, and allow them to send and revieve messages through the network. This is another example of SPMD programming.

The last case is mostly encountered with Master/Worker setups (a specific case of multiple program multiple data) where one program (master) is designed to prepare and distribute work, while another (worker) is designed to perform the work. Only one master is run, but there can be as many workers instances as needed.

Important

The above taxonomy must be take with a grain of salt. Some Master/Worker programs use the network rather than the disk for communication, or use interprocess communication. In that later case, they can only do single-node paralelism. As for Message Passing, it can be done all in a single node. Also shared memory programming can be done on multiple nodes if specific libraries are used.

Slurm tasks

In the Slurm context, a task is to be understood as a running instance of a program which matches the definition of a “process” seen earlier. Tasks for each step are spawned by using the srun command as was explained earlier.

Note

MPI programs can also be started by the mpirun command instead. The mpirun command is provided with the MPI library, while srun is provided by Slurm. There are pros and cons to using one versus another so it will bascally depends on the specific parameters that are needed for starting the program.

The number of tasks (processes) that srun must spawn is decided by the value of the --ntasks option. Each process will be allocated the number of CPUs dictated by the value of the --cpus-per-task option, and the memory obtained by the multiplication of the value of the --mem-per-cpu option by the number of CPUs allocated per task.

Tasks cannot be split across several compute nodes, so all CPUs requested with the --cpus-per-task option will be located on the same compute node. By contrast, CPUs allocated to distinct tasks can end up on distinct nodes.

Each task has an environment variable called SLURM_PROCID set by Slurm to a distinct value from 0 to the number of tasks minus one.

More submission script examples

Here are some quick sample submission scripts. For more detailed information, make sure to have a look at the Slurm FAQ and to follow our training sessions [https://www.ceci-hpc.be/training.html] or to watch the training video [https://www.youtube.com/watch?v=hFsPY0Ti1gE]. There is also an interactive Script Generation Wizard [https://www.ceci-hpc.be/scriptgen.html] you can use to help you in submission scripts creation.

Message passing example (MPI)

#!/bin/bash
#
#SBATCH --job-name=test_mpi
#SBATCH --output=res_mpi.txt
#
#SBATCH --ntasks=4
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=100

module load OpenMPI
srun hello.mpi

Request four cores on the cluster for 10 minutes, using 100 MB of RAM per core. Assuming hello.mpi was compiled with MPI support, srun will create four instances of it, on the nodes allocated by Slurm.

You can try the above example by downloading the example hello world program from Wikipedia [https://en.wikipedia.org/wiki/Message_Passing_Interface#Example_program] (name it for instance wiki_mpi_example.c), and compiling it with

module load OpenMPI
mpicc wiki_mpi_example.c -o hello.mpi

The res_mpi.txt file should contain something like

We have 4 processors
Hello 1! Processor 1 reporting for duty
Hello 2! Processor 2 reporting for duty
Hello 3! Processor 3 reporting for duty

Shared memory example (OpenMP)

#!/bin/bash
#
#SBATCH --job-name=test_omp
#SBATCH --output=res_omp.txt
#
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=100

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun ./hello.omp

The job will be run in an allocation where four cores have been reserved on the same compute node.

You can try it by using the hello world program from Wikipedia [https://en.wikipedia.org/wiki/Openmp#C] (name it for instance wiki_omp_example.c) and compiling it with

gcc -fopenmp wiki_omp_example.c -o hello.omp

The res_omp.txt file should contain something like

Hello World from thread 0
Hello World from thread 3
Hello World from thread 1
Hello World from thread 2
There are 4 threads

Embarrassingly parallel workload example (job array)

This setup is useful for problems based on random draws (e.g. Monte-Carlo simulations). In such cases, you can have four programs drawing 1000 random samples and combining their output afterwards (with another program) you get the equivalent of drawing 4000 samples.

Another typical use of this setting is parameter sweep. In this case the same computation is carried on several times by a given code, differing only in the initial value of some high-level parameter for each run. An example could be the optimisation of an integer-valued parameter through range scanning in a job array:

#!/bin/bash
#
#SBATCH --job-name=test_emb_arr
#SBATCH --output=res_emb_arr.txt
#
#SBATCH --ntasks=1
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=100
#
#SBATCH --array=1-8

srun ./my_program.exe $SLURM_ARRAY_TASK_ID

In that configuration, the command my_program.exe will be run eight times, creating eight distinct jobs, each time with a different argument passed with the environment variable defined by slurm SLURM_ARRAY_TASK_ID ranging from 1 to 8, as specified by the --array parameter.

The same idea can be used to process several data files. To different instances of the program we must pass a different file to read, based upon the value set in the $SLURM_* environment variable. For instance, assuming there are exactly eight files in /path/to/data we can create the following script:

#!/bin/bash
#
#SBATCH --job-name=test_emb_arr
#SBATCH --output=res_emb_arr.txt
#
#SBATCH --ntasks=1
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=100
#
#SBATCH --array=0-7

FILES=(/path/to/data/*)

srun ./my_program.exe ${FILES[$SLURM_ARRAY_TASK_ID]}

In this case, eight jobs will be submitted, each with a different filename given as an argument to my_program.exe defined in the array FILES[]. As the FILES[] Bash array is zero-indexed, the Slurm job array IDs must also start at 0 so the argument is --array=0-7. One pain point is that the number of files in the directory must match the number of jobs in the array.

Note that the same recipe can be used with a numerical argument that is not simply an integer sequence, by defining a Bash array ARGS[] containing the desired values:

ARGS=(0.05 0.25 0.5 1 2 5 100)

srun ./my_program.exe ${ARGS[$SLURM_ARRAY_TASK_ID]}

Here again, the Slurm job array numbering must start at 0 to make sure all items in the ARGS[] Bash array are processed.

Warning

If the running time of your program is small, say ten minutes or less, creating a job array will incur a lot of overhead and you should consider packing your jobs.

Packed jobs example

By default, the srun command in a submission script inherits all non-GRES resource allocated in the job, but with the --exact parameter, you can split the resource and allocate them to multiple steps in parallel.

As an example, the following job submission script will ask Slurm for 8 CPUs, then it will run the myprog program 1000 times with arguments passed from 1 to 1000. But with the -N1 -n1 -c1 --exact option, it will control that at any point in time only 8 instances are effectively running, each being allocated one CPU. You can at this point decide to allocate several CPUs or tasks by adapting the corresponding parameters.

#! /bin/bash
#
#SBATCH --ntasks=8
for i in {1..1000}
do
   srun -N1 -n1 -c1 --exact ./myprog $i &
done
wait

The for-loop can be replaced with GNU parallel if installed on your system:

parallel -P $SLURM_NTASKS srun -N1 -c1 -n1 --exact ./myprog ::: {1..1000}

Similarly, many files can be processed with one job submission script. The following script will run myprog for every file in /path/to/data, but maximum 8 at a time, and using one CPU per task.

#! /bin/bash
#
#SBATCH --ntasks=8
for file in /path/to/data/*
do
   srun -N1 -n1 -c1 --exact ./myprog $file &
done
wait

Here again the for-loop can be replaced with another command, xargs:

find /path/to/data -print0 | xargs -0 -n1 -P $SLURM_NTASKS srun -n1 --exclusive ./myprog

Master/worker program example

#!/bin/bash
#
#SBATCH --job-name=test_ms
#SBATCH --output=res_ms.txt
#
#SBATCH --ntasks=4
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=100

srun --multi-prog multi.conf

With file multi.conf being, for example, as follows

0      echo I am the Master
1-3    echo I am worker %t

The above instructs Slurm to create four tasks (or processes), one running echo 'I am the Master', and the other 3 running echo I am worker %t. The %t placeholder will be replaced with the task id. This is typically used in a producer/consumer setup where one program (the master) create computing tasks for the other program (the workers) to perform.

Upon completion of the above job, file res_ms.txt will contain

I am worker 2
I am worker 3
I am worker 1
I am the Master

though not necessarily in the same order.

Hybrid jobs

You can mix multi-processing (MPI) and multi-threading (OpenMP) in the same job, simply like this:

#! /bin/bash
#
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=4
module load OpenMPI
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun ./myprog

or even a job array of hybrid jobs:

#! /bin/bash
#
#SBATCH --array=1-10
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=4
module load OpenMPI
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun ./myprog $SLURM_ARRAY_TASK_ID

GPU jobs

If you want to claim a GPU for your job, you need to specify the GRES Generic Resource Scheduling [https://slurm.schedmd.com/gres.html] parameter in your job script. Please note that GPUs are only available in a specific partition whose name depends on the cluster.

#SBATCH --partition=PostP
#SBATCH --gres=gpu:1

A sample job file requesting a node with a GPU could look like this:

#!/bin/bash
#SBATCH --job-name=example
#SBATCH --ntasks=1
#SBATCH --time=1:00:00
#SBATCH --mem-per-cpu=1000
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1

module load CUDA

srun ./my_cuda_program

Interactive jobs

Slurm jobs are normally batch jobs in the sense that they are run unattended. If you want to have a direct view on your job, for tests or debugging, you have two options.

If you need simply to have an interactive Bash session on a compute node, with the same environment set as the batch jobs, run the following command:

srun --pty bash -l

Doing that, you are submitting a 1-CPU, default memory, default duration job that will return a Bash prompt when it starts.

If you need more flexibility, you will need to use the salloc [https://slurm.schedmd.com/salloc.html] command. The salloc command accepts the same parameters as sbatch as far as resource requirement are concerned. Once the allocation is granted, you can use srun the same way you would in a submission script.

Further readings

Now that you know how to submit a job, you might be interested in learning more about the following topics:

Slurm : Slurm FAQ, Priority computation, Official documentation [https://slurm.schedmd.com/documentation.html]

Checkpointing: when a job is too long to fit within the maximum wall time of the cluster, it must be checkpointed and restarted. See our training session video [https://www.youtube.com/watch?v=LyWSYYj4zHY&list=PLL1UN3Ywr53_FJWYozVaCbmSJgynibWQ_&index=18] on the subject.

Workflow managers: when you have a multitude of jobs to manage, you might find a workflow manager to be a good helping tool. See the videos of the workshop we organised [https://www.youtube.com/playlist?list=PLL1UN3Ywr5380HTONpdMs09nohnQa6Zzm] on the subject.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Glossary

HPC cluster

A high-performance cluster is a collection of servers, called “nodes” in this context, interconnected by a fast network. A typical cluster consists in one or two management nodes, one or two login nodes (“frontends”), several storage nodes, and a large collection of compute nodes. Users of a HPC cluster must submit jobs to the managing software, called resource manager/job scheduler that decides when and where their computations run.

Compute node

A server in a cluster dedicated to performing the computations. Compute nodes host compute units (processors), volatile memory (RAM), local persistent storage (disk)

Processor

A compute node can host one or multiple processors, fitted in the sockets of the motherboard. These are the chips that perform the actual computation. They are made of multiple cores and caches.

Core

One of the subsystems of a processor that can independently run instructions that are part of a software process. It comprises arithmetic and logic units (ALU) and floating-point units (FPU) and some cache memory and is often split into two hardware threads.

Hardware thread

Independent intruction processing channel inside a core. All threads in a core share some of the computations units of the core. See SMT [https://en.wikipedia.org/wiki/Simultaneous_multithreading] and Hyper-Threading [https://en.wikipedia.org/wiki/Hyper-threading] for more information.

Cache

Small and very fast working memory sitting on the processor, used to hold data transferred to or from the main volatile memory (RAM).

RAM

Acronym for “Random Access Memory”, this is the main working memory of a node, holding the data produced or consumed by the computation process during its lifetime.

Disk

Persistent storage attached to a node. Some disks will hold the operating system (OS), other will hold the user data. Multiple technologies exist: HDD, SSD, NVMe with different capacities/performance/cost tradeoffs.

Operating system

The piece of software that runs the base services on a node and interfaces the hardware resources with the software processes.

Process

Running instance of a program started by the user, the operating system or the job scheduler. A process lives in RAM, can be made of multiple threads, each of which is ideally executed by a core.

Software thread

Execution context (flow) of a sequence of instructions that can be run independently from the rest of the program (the other threads) while sharing the same adress space. Threads in a program are created through mechanisms such as pthreads [https://en.wikipedia.org/wiki/Pthreads] or OpenMP.

Program

Sequence of instructions in either human-readable format (source code) or machine-readable (compiled format (binary). Instructions can refer to mathematical operations, memory transfers, disk access, network connections, etc.

Compiler

Program that produces a binary exectuable from human-readable source code written in a programming language, by contrast with an interpreter.

Interpreter

Program that executes human-readable source code written in a programming language directly by calling the corresponding functionalities in its own code, by contrast with a compiler.

Job

Sequence of instructions treated as a single and distinct unit. More specifically, a job is a sequence of steps, each step consisting in multiple parallel tasks. A job is described by a job submission script and submitted to the job scheduler.

Step

A invocation of a program along with its arguments through the srun Slurm command. The srun command will swpan as many tasks as requested, monitor them, report resource usage, and forward incoming UNIX signals. Each step of a job will have a distinct entry in the accounting for this job.

Task

A running instance of a program started by srun (or mpirun). Multiple tasks of the same step will run in parallel, on possibly distinct nodes and can itself use multiple CPUs in parallel.

CPU

Central Processing Unit. In general, a CPU is synonym to processor, but in this context, it must be understood as a single allocatable unit for computation. Depending on the node configuration and on the parametrization of the job scheduler, it will most often correspond to a core or a hardware thread/

Job submission script

Shell script [https://en.wikipedia.org/wiki/Shell_script] (text file containing shell commands) that describes a job (what commands to run, which program to start, etc.) along with its resource requirements and additional parameters (such as a job name, an email address for notifications, etc.)

Resource requirements

A list of resources that are needed by a job, such as number of CPUs, amount of RAM, possibly GPUs<GPU>, software licences, etc. and a maximum duration for the usage of the those ressources (Wall time limit).

Serial job

Job whose steps consist in only one task using one CPU.

Parallel job

Job whose steps consist in either multiple tasks (distributed-memory job) or a single tasks using multiple CPUs (shared-memory job). How well a job performs as more and more CPUs are made available to it is called scalablilty.

Wall time

The time during which a program runs, from start to finish, “as measured by the clock on the wall”. Jobs on clusters are subject to maximum walltime to allow fair sharing of the cluster.

Job scheduler

Piece of software that accepts, schedules and dispatches, jobs to resources based on their resource requests and priority.

Scheduling

Action of deciding which job can use which resources and when, based on resource availability, job priorities and backfill opportunities. Jobs that can currently use resources are said to be running, the others are pending.

Job priority

Number associated with a job that is used to decided the ordering in which jobs are considered for resource allocation. The priority can be computed based on multiple criteria such as fairshare, job size, queue time, quality of service, etc.

Backfill

Scheduling policy by which a job with a lower priority can start before a job with higher priority, provided it does not delay that higher-priority job’s start time. This can be either because it requires a completely distinct set of resources, or because it will free the resources before the predicted time of the higher-priority job.

Fairshare

Fairshare is a measure of how far away from a fair usage of the resources a user or an account is, based on its cluster share. Users who used the cluster a lot in the recent past (i.e. have consumed many CPU.hours) have a lower fairshare than users who did not. Fairshare is adapted in realtime as the running jobs consume resources.

CPU.hour

Unit of computing resource usage corresponding to using a full CPU for one hour, or, equivalently, two CPUs for half an hour, etc. The concept is similar for node.hour, node.day, GPU.hour, etc.

Account

A Slurm account is an administrative concept that allows tracking and organising resource usage among user organisation levels (departments, units, etc.)

Share

Portion of the total available resource a cluster can offer that is “promised” to a user or an account, often based on administrative concerns.

Quality of service

Functionality by which a user can request particular privileges for a job, such as for instance a priority boost.

Shared-memory job

Job whose parallel threads or processes can all address the same memory, or at least a common portion of memory. Such jobs can only run on a single node.

Distributed-memory job

Job whose parallel tasks each have their own memory space. Such jobs can spread across multiple nodes provided they rely on a mechanism for data communication between tasks, which can be through the network XXX or via the disks.

Shared-memory programming

Parallel programming pardigm where all theads or process are able to read and/or write in the same memory space, for instance with threads spawed from the same process, or processes using a shared memory segment. Shared-memory programming is typically done in a shared-memory job, but can also happen in a distributed-memory job provided a PGAS <https://en.wikipedia.org/wiki/Partitioned_global_address_space>`_ library such as Unified Parallel C [https://upc.lbl.gov], Coarray Fotran [https://en.wikipedia.org/wiki/Coarray_Fortran] or OpenSHMEM <http://openshmem.org/site/> is used.

Distributed-memory programming

Parallel prgramming paradigm where processes exchange messages rather than sharing memory. If the exchange mechanism is able to travel through networks, as is the case with MPI, the processes can live in distinct nodes, if it is not the case, such as with named pipes [https://en.wikipedia.org/wiki/Named_pipe] or UNIX sockets [https://en.wikipedia.org/wiki/Unix_domain_socket]

OpenMP

Standard for shared-memory programming abstracts explicit threads and offer parallel programming constructs such as a parallel for loop. Many optimised libraries are built using OpenMP.

MPI

Message Passing Interface. De-facto standard for distributed-memory programming. Multiple implementations exist for various languages, some are open-source, some are sold by hardware manufacturers or compiler vendors.

GPU

Graphical Processing Unit. Hardware component initially designed to connect to the computer screen, later evolved to off-load heavy computations from the processor. The compute power of a GPU is much larger than that of a processor, but the way it is programmed is very different. Specific libraries such as CUDA [https://en.wikipedia.org/wiki/CUDA], ROCm [https://en.wikipedia.org/wiki/ROCm], OpenACC <https://en.wikipedia.org/wiki/OpenACC>, OpenCL [https://en.wikipedia.org/wiki/OpenCL] must be used.

Optimised library

Software library that can be included into a program and offer access to optimised linear algebrae, signal processing, statistical, etc. functionalities. Open-source examples are OpenBLAS [https://en.wikipedia.org/wiki/OpenBLAS], BLIS [https://en.wikipedia.org/wiki/BLIS_(software)], or Atlas [https://en.wikipedia.org/wiki/Automatically_Tuned_Linear_Algebra_Software]. Commercial libraries include for instance MKL [https://en.wikipedia.org/wiki/Math_Kernel_Library]. Like compilers and other software, they are most often organised in environment modules, or system modules

Environment module

Mechanism that allows modifying the environment (environment variable, alias, etc.) in which programs are started by the shell. By modifying variables such as $PATH or $LD_LIBRARY_PATH, users can choose which of the installed software they want to use. Modules are often organised into releases.

Release

In the context of modules, a release is a set of modules related to software whose versions and toolchain have been carefully chosen so as to be compatible.

Toolchain

Set of compiler, optimised libraries and MPI modules that are used together to compile software.

Shell

Command-line interface to the operating system that interpret user commands such as starting a program, copying a file, etc. Examples include Bash and Zsh.

CLI

Command-line interface. User interface that involves the keyboard to enter command in response to a prompt in a terminal window, by contrast with a TUI (text user interface), that lets users interact with menu items or widgets in a terminal through the keyboard or a GUI (grahipcal user interface) that allows users to click on buttons and menu items with the mouse.

Terminal

Piece of software that manages the interactions between the shell and the console of the user. The shell can be local to the machine or running on a remote server and connected to through SSH.

Console

Computer equipment consisting in a keyboard and a screen.

SSH

Secure Shell. Protocole by which users can connect to remote servers (for instance a login node of a cluster) using their login and a combination of authentication factors <https://en.wikipedia.org/wiki/Multi-factor_authentication>_ (password, key, hardware token, etc.).

File

A piece of data (text, number sequence, image, etc.) stored in a filesystem and primarily identified by its name. Along the name, other metadata information is stored by the filesystem such as owernship, permissions, or creation date. Files are organised in a hierarchy of directories

Directory

Cataloging structure that contains files or to other directories. Also called folder in some contexts. In HPC cluster, each user has a home directory where they can write files and create sub-directories. It is by default their current working directory when they connect to the cluster.

Filesystem

Data structure and assorted mechanism to store and retrieve files. A filesystem local to a (compute) node can only be accessed from that node. By contrast, a network filesystem (e.g. NFS [https://en.wikipedia.org/wiki/Network_File_System]) is hosted on a (storage) node and exported to all other noeds, and a parallel filesystem (e.g. BeeGFS [https://en.wikipedia.org/wiki/BeeGFS]) is hosted on multiple (storage) nodes and exported to all other nodes.

File permission

Meta data associated with a file, that defines what type of access (read, write, execute) the owner, its group, and the others, have on the file.

Profiler

Software that analyses other software to find out which functionality or operation takes the most time.

Debugger

Software that helps analysing other software to find bugs and problems.

Environment Variables

A shell variable that holds values that can be specified by the user, by the environment module system, or by Slurm, that can alter the way sofware behaves.

Scalability

How well a job or program can use increasing amounts of computing resources. A program that strongly scales takes less and less time as more computing power is used. A program that weakly scales takes roughtly the same time when more data and more computing power are used in the same proportions.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Disk space

Danger

There is no backup of the data stored on any cluster. Any removed file is lost for ever. It is the user’s responsibility to keep a copy of the contents of their home in a safe place.

Each cluster is equipped with several file systems [http://en.wikipedia.org/wiki/File_system] that can be used to store files. These disk spaces have different properties and each of them is designed to best fit different usage intents. They are listed in the table here-under.

Disk space | Scope | Environment variables (depends on cluster) | Lifetime
Home | cluster | $HOME | cluster or account lifetime + grace period [https://www.ceci-hpc.be/faq.html#3.3]
Workdir | cluster | $GLOBALSCRATCH | one year
Local scratch | node | $TMPDIR,$LOCALSCRATCH | job lifetime
Global Home | CÉCI | $CECIHOME | funding or account lifetime + grace period [https://www.ceci-hpc.be/faq.html#3.3]
Transfer | CÉCI | $CECITRSF | as short as needed
Long-term | external |  |

The ‘Scope’ column indicates from where the disk space is accessible. A scope of ‘cluster’ means all compute nodes and frontends in the cluster share the same filesystem. By contrast, a scope of ‘node’ refers to storage space that is distinct for each node ; it is local to that node and cannot be accessed from outside the node. The ‘CÉCI’ scope means the filesystem is accessible from all compute nodes and all frontends of all CÉCI clusters. Finally, a scope of ‘external’ refers to machines that are outside the perimeter of the CÉCI consortium and are managed by the universities.

The ‘Lifetime’ column indicates the duration for which the data will be kept. Data residing in the home directories will reside there as long as the cluster is in production and the account owning the data was renewed no later than a couple years ago (grace period [https://www.ceci-hpc.be/faq.html#3.3]). When a date for decomissioning is decided, users are warned, and instructions are offered to migrate the data either to a newer cluster, or to another storage.

Warning

Data belonging of accounts that are expired for more than 500 days are eligible for deletion, after notification to the data owner, or their supervisor should the owner be unreachable.

When a date for purging old data is reached, the past owner is contacted if possible, or their supervisor to warn about the data removal. Data in Workdirs/Global scratches are often cleaned during maintenance operations, while data in Local scratches are cleaned after each job. Data in Global home will remain as long as there is sufficient funding to maintain and replace the infrastructure when needed. Finaly, data in Transfer should be removed as soon as tranfers are completed.

Note

You can also check the presentation given on our yearly training sessions about using the different storage solutions on the clusters [https://indico.cism.ucl.ac.be/event/110] . On the final slides is pointed where to find some slurm submission scripts examples.

Home file system

Upon login on a login node or front-end, you will be end up in your home directory. This directory is available on the front-end and all compute nodes of a cluster. Its full path can be shown with echo $HOME and you can return there with a simple cd command.

The home filesystem is dedicated to source code (programs, scripts), configuration files, and small datasets (like input files.)

Do not use this area for your main working activities; use your workdir directory instead (see next section).

Note

Quotas and time limits are enforced on the use of space on the HOME directories. See section “Quota” for details.

Workdir file system

The workdir or globalscratch is a high-performance shared disk space common to all compute nodes and to the front-end of a cluster. Its full path can be shown with echo $GLOBALSCRATCH. The path will differ from one cluster to other. The globalscratch is often built using a fast parallel filesystem such as Lustre [http://en.wikipedia.org/wiki/Lustre_(file_system)], GPFS [http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System], or FraunhofferFS [http://en.wikipedia.org/wiki/FhGFS].

The workdir should be used to store the files generated by your batch jobs. You can also copy there large input files if required. After that, it is customary in the job script to create a subdirectory with the job id, where all temporary data will be written, and to clean that directory when the job finishes, and after having copied results to another location either on the same filesystem if the results are to be consumed by a later job, or on another filesystem such as the home or a remote long-term storage.

To use a globalscratch, you might first need to create a directory by yourself. It is then common to name it after your login. On some clusters, it might already be created for you. Anyway, it is always safe to try to create it on any CÉCI cluster with the command.

mkdir -p $GLOBALSCRATCH

Note

Some clusters have a quota on the use of the workdir directories. See section “Quota” for details.

Warning

The data in the globalscratch directory can be removed at any time specialy during maintenance periods.

Local Scratch file system

The local scratch is the temporary disk space available on all compute nodes and is only visible from within the compute node it belongs to. On the CÉCI clusters, they are available through the $LOCALSCRATCH or $TMPDIR environment variable. Often they are built on top of a fast, but redundancy-lacking, RAID-0 system.

There you can write/read temporary results during your job, copy the results of interest back to the home directory, and then delete it at the end of the job script. Example commands can be found in F.A.Q.-Q11 section

Note

Files stored in the scratch directory on each node are removed immediately after the job terminates. You will not be able to access files in the scratch directory after your job has completed. Furthermore files in scratch directory are not accessible from any other nodes (compute node and login nodes). Therefore, all files you want to save must be copied from the scratch directory to your home as part of your job.

Using the scratch directory when running a batch job is often more efficient than using the home or workdir. If your jobs performs a lot of disk I/O to files that does not need to be shared between nodes, then please use this directory. This relieves the load on the central disk servers, and most of the time also makes your job run faster.

Warning

There is not quota limit on the local scratch. The user has to be careful not to fill the space otherwise the job will probably crash.

The scratch size depends of the node type. For example on Hercules:

Node type | Size
Dell M610 | 600 GB
Dell M610x | 600 GB
HP DL360 | 600 GB
HP SL230 | 1.2 TB

CECI’s Common filesystem

A detailed information about Global Home and Transfer filesystem is available on the common filesystem section.

Long-term storage file system

Long-term storage or archive is also built on stable technologies, but quota there are less limited. The downside is that the long-term storage is often not directly connected to the cluster so it can be quite slow. It may also not be free of charge. CECI does not provide long-term storage. Ask the systems administrator at your institution which kind of long-term storage they offer. Then you ca transfer your data as explained in the file transfer section.

Final remarks

As scratch spaces are not meant to store data in the long term, you should expect them to be cleaned automatically after some time. This does not mean that you should not do it yourself though: always clean up after your job. This is especially important after a job crashes before the cleaning operations in the submission script had a chance to run.

You need to be careful about filenaming when using a parallel job to avoid overwriting files by two different parts (threads, or processes) of the program, especially on global filesystems.

Even if your data is small and the quota allow you working in your home directory, you should consider using scratch spaces as they are in general much faster. The global filesystems are much faster because they handle requests and store data in parallel, while the local filesystems are faster as they do not need to access the network and they are used by far less jobs at a time.

Quota

Quotas are setup on your home directory and in some clusters on your workdir space.

The quota system will be based on soft quota, hard quota and a grace period.

A soft quota is the disk space level below which nothing happens for the user. When your reach the soft quota, a warning email is sent to you on some systems. On others, you get a message when you connect. This does not prevent you writing more data until the grace period is reached.

The grace period is the amount of time during which you are allowed to go over the soft limit before the system blocks your ability to write more data.

The hard quota is the disk space level above which the system blocks immediately your ability to write more data. However, you will still be able to read your data. If your hard quota is exceeded or if you exceed the grace time for your soft quota, your will be blocked from submitting jobs until you clean your space.

The table below present the storage quotas for CÉCI user accounts on each cluster. Use the command ceci-quota to get all your current quotas usage.

Cluster
file system
env variable
Soft quota
Hard quota
Grace

NIC5

home

workdir

$HOME

$GLOBALSCRATCH

100 GB

5 TB

110 GB

5 TB

Dragon2

home

workdir

$HOME

$GLOBALSCRATCH

40 GB
44 GB

Dragon1 [1]

home

workdir

$HOME

$GLOBALSCRATCH

20 GB

20 GB

23 GB

23 GB

Hercules

home

workdir

$HOME

$GLOBALSCRATCH

200 GB

400 GB

1 TB

4 TB

9 weeks

1 weeks

Lemaitre3

home

workdir

$HOME

$GLOBALSCRATCH

100 GB

none

100 GB

none

n/a

Nic4

home

workdir

$HOME

$GLOBALSCRATCH

20 GB

none

25 GB

none

All Clusters

Global Home

Transfer

$CECIHOME

$CECITRSF

100GB

1 TB

120 GB

10 TB

10 days

[1]
Clusters where $HOME and $GLOBALSCRATCH are the same filesystem so the quota is applied on both.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Transferring files to and from the clusters

This section is about transferring data between your computer and a cluster. To transfer data between cluster, see the section trsf in Using the common filesystem

Those examples are only for Linux and MacOs computers or if you are using WSL. The commands are executed in your computer. Your SSH client configuration file must be correct as explained in the Connecting from a UNIX/Linux or MacOS computer section. Replace cecicluster by the Host name of the cluster defined in your .ssh/config file.

Copying a file or directory

The simplest way to copy a file to or from a cluster is to use the scp command.

scp ./file.txt cecicluster:destination/path/

Copying it back is done with

scp cecicluster:path/to/file.txt .

If you want to copy a directory and its content, use the -r option, just like with cp.

scp -r cecicluster:path/to/folder .

Transferring a large number of small files

Transferring a lot of small files will take a very long time with scp because of the overhead of copying every file individually. In such case, using the tar command [http://www.linfo.org/tar.html] will reduce the transfer time significantly. You can first create a tar archive, then scp it as a single file and then ‘untar’ the file. But the most efficient way is to do all three operations in one go, without creating an intermediate file, like this:

tar cz ./source_dir | ssh cecicluster 'tar xvz -C destination/path'

This will create a large file containing the small files and remove the overhead of dealing with many small files.

Copying it back from cluster to your computer is done with:

ssh cecicluster 'tar -C source/path -cz source_dir' | tar -xz

Transferring large files

When transferring large files, it is often interesting to use the -C option of scp to first compress the file, send it, and then decompress it. Using it simply with

scp -C ./large_file.txt cecicluster:destination/path/

Resuming interrupted transfers

If, for any reason, a transfer is interrupted, you might end up with part of the files being transferred. Rather than restarting the transfer from scratch, you should then use the rsync [https://en.wikipedia.org/wiki/Rsync] command. The rsync command will compare the source and destination directories and only transfer what needs to be transferred: missing files, modified files, etc.

Use it this way (assuming again that your SSH client is properly configured):

rsync -va ./source_dir cecicluster:destination/path

Make sure not to leave trailing slashes in your path names (e.g. NOT destination/path/) as you might end up with a full copy of the directory inside the existing, partial, one. Use the -n (dry-run) option of rsync to check what will happen before you run the actual command.

If one large file is left half-transferred, you can resume it using the --partial.

Transferring code

Source code is a specific type of data and should be treated as such. The best way to transfer code from one computer to another is to host the code in a source code repository using a versioning system such as git [http://en.wikipedia.org/wiki/Git_(software)] (more common) or mercurial [http://en.wikipedia.org/wiki/Mercurial] (easier to use) and clone the repository from your laptop to the cluster.

Synchronising with a local directory

If you want to keep two directories (one on your laptop, and one on the cluster) in sync, you can do that with rsync using its --delete option. But that is only one-way so you need to really think in what direction you do it, and it does not scale beyond two synchronized directories.

A real option is to use Unison [http://en.wikipedia.org/wiki/Unison_(file_synchronizer)], a piece of software that can detect and handle conflicts (incompatible changes made to the same file in the two directories that must be kept in sync.)

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Using the common filesystem

All CÉCI clusters are connected to a central storage system that is visible to all compute nodes of all clusters. This system runs on a fast, dedicated, network. It will become the home of the users in the near future, but in this first phase, it is set up as an additional home besides the default, cluster-specific, home.

This storage system is installed at two CÉCI locations (at ULiège and UCL) and data are replicated synchronously on both locations to ensure data safety and a certain level of high availability. Moreover, on each site, a local cache is setup to mask the latencies of the network and make sure the user experience is as smooth as possible. Those caches are replicated asynchronously with the central storage, meaning that files that are written there will appear after some delay on the other clusters.

Note

There is no point in creating a file from a cluster and then quickly going to another cluster and running ls rapidly waiting for the file to appear. The file will be copied on the main storage only after 15 seconds, and the metadata will be copied on the other clusters after up to 60 seconds.

It also means that if you modify the same file from two different clusters, the result is undefined.

Warning

Do not write to the same file from two different clusters at the same time. This would corrupt the file.

After some delay, the files which are not read from a cluster are removed from the cache connected to that cluster. The metadata are kept, meaning that the files appear when you use ls but the data are not. The data will be synced back as soon as you try to access the file.

The common storage is split into four distinct directories:

Home

Trfs

Proj

Soft

You, as a user, have read and write access to the first two, maybe to the third one, and only read access to the fourth one.

Note

If you run the du command on a cluster, you will only get the disk usage on the cache, not on the main central system! To find out the used space on the common filesystem folders always add the --apparent-size option:

du --apparent-size -sh  /common/storage/path

For more details on this topic keep reading the following section.

Home

The /CECI/home partition is designed to work as the current home filesystems you have access to on the clusters. You have a 100GB quota for you to store your own software, configuration files, and small (input) data files or (output) result files. The advantage of using this home rather than the cluster-specific one is that no matter how you configure your environment on one cluster, you will find the same environment on all clusters without the need to move and copy files around.

Note

The 100GB quota is on the local caches connected to the clusters, But your quota on the central storage is NOT 5 x 100GB. Do not try to play the system and fill your quota on all the clusters at the same time ; you might be able to go beyond the quota limit, but it could result in file corruption and loss of synchronisation.

To get your current usage on the main central system, use the quota command from one of the clusters. If the quota command does not list the central storage, try first listing its contents with for instance ls $CECIHOME. It should then appear.

In the long term, this filesystem will be the default home on the clusters, but at the moment, you will need to move there (using cd) explicitly. An environment variable named $CECIHOME points to your home on the common filesystem so you can issue cd $CECIHOME.

Note

If you have, in your CÉCI home, programs that you compiled by yourself, make sure you have used the tips and tricks mentioned in Compiling for multiple CPU architectures to make sure they will be able to run on every CPU architecture present in the CÉCI clusters.

To make clearer the previous explanations about the complex solution running the common storage, we make a special consideration about the usage of the du command to monitor the space on disk taken by files and directories.

When copying, for example, a 6MB file from one of the cluster’s home folder to $CECIHOME and perform a du -sh you will obtain:

mylogin@nic4: $ cp ~/file.dat $CECIHOME/
mylogin@nic4: $ du -sh $CECIHOME/file.dat
6.0M   $CECIHOME/file.dat

In order to avoid the inherent problems due to wide-area interconnects latencies, the rest of the caches at the different clusters, are served only the metadata of the files created on one of them. That is to say, your file.dat file will be actually copied, synced and stored on both main storages at ULiège and UCL, but the rest of the caches will only have the information that the file exist on the common space, the actual contents will be transferred only on demand.

Then, if after the previous steps you login on another CÉCI cluster and perform du -sh you will see:

mylogin@dragon1: $ du -sh $CECIHOME/file.dat
0     $CECIHOME/file.dat

If in this cluster you access the file, i.e. open with an editor, do a cat or less, etc. then the actual data contained on the file will be transferred from one of the main storages to the cache on this cluster, afterwards you should get:

mylogin@dragon1: $ du -sh $CECIHOME/file.dat
6.0M   $CECIHOME/file.dat

To summarize, you should never rely on du -sh to verify if a file is properly stored or copied, and this is valid in general for any kind of filesystem. A more appropriate action would be to check for file consistency with a hash tool and verify to get the same output on different clusters:

mylogin@dragon1: $ md5sum $CECIHOME/file.dat
da6a0d097e307ac52ed9b4ad551801fc  $CECIHOME/file.dat

In the case that you want to know which is the approximate space taken by a file or directory on $CECIHOME then add the --apparent-size option to du:

mylogin@dragon1: $ du -sh --apparent-size $CECIHOME/file.dat
6.0M   $CECIHOME/file.dat

the output should be closely the same on all the clusters, independently if files were accessed or not. Notice that the man page for du defines the tool as du - estimate file space usage.

In case this info is very different among clusters, please submit a ticket on the CÉCI Support page [https://support.ceci-hpc.be/cecihelp].

Trfs

The /CECI/trsf partition is meant for large file transfers from one cluster to another. The user quota on that partition is 1TB soft and 10TB hard with a grace period of 10 days. This means that you use up to 10TB, but as soon as you go above the 1TB limit, you will have 10 days to remove files and come back below the 1TB limit. If you fail to do so, you will not be able to write to the disk anymore until you comply. This directory is purged of old files regularly, each time the purge is performed all files created earlier than 6 months ago will be irremediably removed.

Note

Do not use that partition as a storage partition. It is meant for transfers and can only hold temporary data.

This partition can be used to transfer large files from the scratch space (see globalscratch) of a cluster to the scratch space of another one. It will be much faster than using scp or rsync directly. As this filesystem is optimized for large files, if you have a lot of small files, you must tar them otherwise you will experience very poor performances. For instance, assuming you have a directory transfer_me on the scratch of Lemaitre2 and want to move it to the scratch of NIC4:

On Lemaitre2:

cd $GLOBALSCRATCH
tar cvzf $CECITRSF/transfer_me.tgz ./transfer_me

On NIC4, a bit later:

cd $GLOBALSCRATCH
tar xvzf $CECITRSF/transfer_me.tgz

Everyone can write into the $CECITRSF directory, but the sticky bit is set so that you can only delete your own files.

Note

Even if this is a two-step operation, if will be much faster than using scp for large files as it benefits from the fast dedicated network. But you still have the option of using scp of course.

An environment variable named $CECITRSF points to that location.

Proj

The /CECI/proj partition will be dedicated to projects, i.e. groups of researchers working towards a common goal and sharing files. Projects will be created upon motivated request, with a specific quota and a specific duration.

Note

Currently, if you want to enable a project for your group please contact ceci-logist@lists.ulg.ac.be to do the request.

Soft

The /CECI/soft partition will hold all the software that will be available on all clusters. They will be compiled for each processor type that is present in one of the CÉCI clusters. On each node, the version specific to the hardware of the node will be selected by the module.

Behind the scenes

The CÉCI shared storage is based on two main storage systems hosted in Liège and Louvain-la-Neuve using the GPFS filesystem developed by IBM [https://www.ibm.com/support/knowledgecenter/SSFKCN/gpfs_welcome.html]. Those two storage systems are synchronously mirrored, meaning that any file written to one of them is automatically written to the other one.

In addition, to keep the data on the common partitions synchronized among the different geographical locations of the CÉCI clusters, it is used the Active File Management (AFM) implementation [https://www.ibm.com/support/knowledgecenter/SSFKCN_3.5.0/com.ibm.cluster.gpfs.v3r5.gpfs200.doc/bl1adv_afm.htm]. Within this setup a global namespace [https://www.ibm.com/support/knowledgecenter/SSFKCN_3.5.0/com.ibm.cluster.gpfs.v3r5.gpfs200.doc/bl1adv_afmglobalnamespace.htm] is defined to share a single filesystem among different gateway nodes which are physically deployed on each of the CÉCI universities.

These five smaller storage systems that serve as buffers/caches and the main ones are interconnected through a dedicated 10GBps network. Those caches located on each site and are tightly connected to the cluster compute nodes.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Sharing files among CÉCI users

As scientists work in teams, it is often necessary to share files (be it data or code) among several CÉCI users. Access to files is governed by the UNIX file permission system so you will need to get familiar with it to share files. A general introduction can be found on Wikipedia [https://en.wikipedia.org/wiki/File_system_permissions]. Actually, sharing files with every one else is easy. It is when you need to give some people access to you files and not to others that things become a bit more convoluted.

Sharing with every one is easy: you simply need to set the correct UNIX permissions on your home directory and the files/directories you want to share. Sharing with only certain people through hiding is also easy (access by unauthorized parties is possible but improbable). Sharing with only certain people through real enforcement requires our intervention (access by unauthorized parties is then impossible).

Sharing with everyone

To share with everyone you need to make sure that your home directory has read and execute permission for others (e.g. rwxr-xr-x or 755 in octal form). Then, everyone can list the content of your home directory and access any file in there that has read access for others. The reasoning applies recursively to sub-directories.

To find which permissions apply to your home directory, run the ls command (remember that ~ is the home directory of the current user, the -l options is used to show permissions, the -d option is to list the directory itself rather than its content, and the -L option is necessary because on most CÉCI systems, the home directory is a symbolic link):

ceciuser1@cecicluster:~ $ ls -dlL ~
drwxr-xr-x 56 ceciuser1 ceciuser1 8192 Aug 12 16:20 /home/ceciuser1

Here, the home directory of the ceciuser1 user on cecicluster is readable and accessible by everyone.

Should the above command return something like the following:

ceciuser1@cecicluster:~ $ ls -dlL ~
drwxr-x--- 56 ceciuser1 ceciuser1 8192 Aug 12 16:20 /home/ceciuser1

another user (for instance here ceciuser2) would not be able to see the contents:

ceciuser2@cecicluster:~ $ ls -l ~ceciuser1
ls: cannot open directory /home/ceciuser1: Permission denied

Then the chmod command must be used to give access to others. The manual of the chmod command can be found by issuing the man chmod command.

ceciuser1@cecicluster:~ $ chmod o=rx ~
ceciuser1@cecicluster:~ $ ls -dlL ~
drwxr-xr-x 56 ceciuser1 ceciuser1 8192 Aug 12 16:20 /home/ceciuser1

The above chmod command reads, in plain English, Change the permissions of my home directory so that others have (=) read and execute access to it., so that other users will be able to see its contents. Indeed, user ceciuser2 now can list the content of the home directory of user ceciuser1.

ceciuser2@cecicluster:~ $ ls -l ~ceciuser1
drwxr-xr-x ceciuser1 ceciuser1 Aug 12 16:20 shared_dir
drwxr-x--- ceciuser1 ceciuser1 Aug 12 16:20 private_dir
-rw-r--r-- ceciuser1 ceciuser1 Aug 12 16:20 fileA
-rw-r----- ceciuser1 ceciuser1 Aug 12 16:20 fileB

What we see is that other (all other) users have read access to shared_dir and fileA, but not to fileB for instance:

ceciuser2@cecicluster:~ $ cat ~ceciuser1/fileA
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
ceciuser2@cecicluster:~ $ cat ~ceciuser1/fileB
cat: /home/ceciuser1/fileB: Permission denied

Restricting access through hiding

Sometimes you do not want to share your files with all other users, though.

To share with only a restricted list of users, one option is to remove the read permission on your home directory. Then, other users won’t be able to list the content of your home directory, but if they know exactly the name of the file they need, and that file has read permission for others, they will be able to read it. If the name is not trivially guessable, that file is actually hidden but accessible to those who know it exists.

To reach this state, use the chmod command this way:

ceciuser1@cecicluster:~ $ chmod o=x ~
ceciuser1@cecicluster:~ $ ls -dlL ~
drwxr-x--x ceciuser1 ceciuser1 Aug 12 16:20 /home/ceciuser1

We see that the permissions are now drwxr-x–x rather than drwxr-xr-x, and other users can’t see what’s inside the directory:

ceciuser2@cecicluster:~ $ ls -l ~ceciuser1
ls: cannot open directory /home/ceciuser1: Permission denied

Nevertheless, if ceciuser2 knows that ceciuser1 has a file named fileA, in his home directory, he can read its contents.

ceciuser2@cecicluster:~ $ cat ~ceciuser1/fileA
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

Using this settings, ceciuser1 can decide with whom he shares a file. Setting the file or directory name to an unguessable character string, and giving that string to ceciuser2, is roughly equivalent to giving ceciuser2 a password to access the file. Note though that anyone who knows the ‘password’ (i.e. the secret file/directory name) will be able to access it.

ceciuser1@cecicluster:~ $ ls -l ~
drwxr-xr-x ceciuser1 ceciuser1 Aug 12 16:20 shared_dir_&aqw1AQW
drwxr-x--- ceciuser1 ceciuser1 Aug 12 16:20 private_dir
-rw-r--r-- ceciuser1 ceciuser1 Aug 12 16:20 fileA
-rw-r----- ceciuser1 ceciuser1 Aug 12 16:20 fileB

Everyone who knows about shared_dir_&aqw1AQW will be able to access it but it won’t be discoverable by others.

Enforcing the restricted access (one to many)

The above solution is sufficient in most cases. But if you want to guarantee that only a specific set of users have access to your file even if they know its name, the CÉCI system administrators need to be involved.

To share with only a limited restricted list of users in a ‘one to many’ fashion (for instance you are installing software for all people in your research group, or you have all your group’s data in your home directory), and enforce that others cannot access the data, one option is to add each authorized parties to your group. Then, the above reasoning applies to the members of your group provided you replace every occurrence of others with group in the above paragraphs. Users not in your group will not be able to access your data at all (as long as your home directory has no execute permission for others of course.)

Assuming the CÉCI system administrators have included ceciuser2 in ceciuser1‘s group:

ceciuser2@cecicluster:~$ id
uid=3000014(ceciuser2) gid=3000014(ceciuser2) groups=3000014(ceciuser2),3000003(ceciuser1)

then ceciuser2 has access to any file/directory that is group-readable. Beware, that means most often all of ceciuser1‘s files under the default settings!

Enforcing the restricted access (many to many)

To share with a larger group, allow everyone in the group to share as well, and still enforce that others cannot access the data, the most convenient option is to create a UNIX group specifically for that group. Then, the reasoning in the previous paragraph applies provided you first change the group to which the data belong.

Assuming the CÉCI system administrators have created a group ourgroup with ceciuser1 and ceciuser2,

ceciuser1@cecicluster:~ $ getent group ourgroup
ourgroup:*:4999998:ceciuser1,ceciuser2

then ceciuser1 can share the shared_dir directory with chgrp

ceciuser1@cecicluster:~ $ chgrp ourgroup shared_dir
ceciuser1@cecicluster:~ $ ls -l ~
drwxr-x--- ceciuser1 ourgroup Aug 12 16:20 shared_dir
drwxr-x--- ceciuser1 ceciuser1      Aug 12 16:20 private_dir
-rw-r--r-- ceciuser1 ceciuser1      Aug 12 16:20 fileA
-rw-r----- ceciuser1 ceciuser1      Aug 12 16:20 fileB

Note that the permissions of shared_dir allow here read access to shared_dir by a member of the ourgroup group, but no write access, and also no read access to users not in the ourgroup group. Should ceciuser1 want other users in the group to be able to write files in his directory, that can be done easily:

ceciuser1@cecicluster:~ $ chmod g+w shared_dir
ceciuser1@cecicluster:~ $ ls -l ~
drwxrwx--- ceciuser1 ourgroup Aug 12 16:20 shared_dir
drwxr-x--- ceciuser1 ceciuser1      Aug 12 16:20 private_dir
-rw-r--r-- ceciuser1 ceciuser1      Aug 12 16:20 fileA
-rw-r----- ceciuser1 ceciuser1      Aug 12 16:20 fileB

The above chmod command reads, in plain English, Add (+) write permission to members of the group owning shared_dir

Concluding remarks

UNIX groups corresponding to TIER1 project are also created on the CÉCI clusters and can be used there.

on the clusters, the umask – the configuration setting that dictates the permissions set to newly created files – is set to 0002 or 0022, meaning that by default, your files are created with permission 664 or 644 respectively.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Copy files and folders to the /CECI/proj partition

Introduction

In Linux, the users has a user identifier uid, a group identifier gid and a list of other groups to which the user is a member. In the CÉCI clusters the uid and gid are equal to your login account.

user@cluster:~$ id
uid=NNNNNN(user) gid=NNNNNN(user) groups=NNNNNN(user),JJJJJ(ceci_group),KKKKK(group2), ...

The files and folders have a user owner and a group owner. When you create or copy files, they get your uid and gid

user@cluster:~$ mkdir test
user@cluster:~$ touch test/file{1..3}
user@cluster:~$ ls -l test/file*
-rw-rw-r-- 1 user user 0 May 28 10:02 test/file1
-rw-rw-r-- 1 user user 0 May 28 10:02 test/file2
-rw-rw-r-- 1 user user 0 May 28 10:02 test/file3

CÉCI users can be part of a project which share files in a group folder inside /CECI/proj partition. All the files and folders inside the /CECI/proj/ceci_group folder must have the group name equal to the name of CÉCI group you belong instead of the name of the user.

Copying from a cluster front-end to /CECI/proj

If you want to copy files from your HOME, the GLOBALSCRATCH or other cluster location to the directory of your group inside proj partition you first need to change your gid with newgrp command, copy your data and exit to restore your gid.

user@cluster:~$ newgrp ceci_group
user@cluster:~$ id
uid=NNNNNN(user) gid=JJJJJ(ceci_group) groups=NNNNNN(user),JJJJJ(ceci_group),KKKKK(group2), ...
user@cluster:~$ cp -r folder /CECI/prog/ceci_group/path/.
user@cluster:~$ exit

Replace ceci_group with the name of the CÉCI group you belong.

Copying from a your computer to /CECI/proj

To copy the data form your Linux or MacOS computer you must add to your rsync [https://en.wikipedia.org/wiki/Rsync] command the -og and --chown=user:ceci_group flags. For example, to copy a local folder inside /CECI/proj/ceci_group/path/ folder using one of the clusters [https://www.ceci-hpc.be/clusters.html] front-end, you can use the command:

local_user@mycomputer:~$  rsync -azv -og --chown=user:ceci_group local/folder cluster:/CECI/proj/ceci_group/path/.

Where cluster is one of the host defined in your ssh configuration file [https://www.ceci-hpc.be/sshconfig.html], user is your CÉCI user identifier and ceci_group is the name of the CÉCI group you belong.

Note

If the rsync version in MacOS is to old (< v3.1.0) and --chown option is not available, the newer version must be installed, for instance with Homebrew [https://brew.sh/].

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Making your files Secure

By default, the permissions on files on a UNIX system are rather lax. Everyone can see your files, so it is your responsibility to hide or protect files you do not want others to access.

You can use the chmod go-rw <filename> command file per file, or use the umask 0077 command in your Bash startupfile (.bash_profile) so that all files that are created are only visible by you.

Alternatively, you can change the permissions on your home directory directly: chmod og-rx ~.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Making your files safe

Even though your data on the CÉCI clusters are stored on arrays of redundant disks (RAID), to make them safe under some hardware failure scenarios, they are not safe against accidental deletions.

Some possible options you can take to protect your files from the latter events are:

keep a synchronized copy of your clusters home on your laptop, using for instance rsync or unison;

promptly copy output data from the cluster scratch space to mass storage;

use versioning for all your text files.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Long term data storage

The CÉCI does not currently offer long-term data storage. Each cluster has a home filesystem dedicated to long-term storage code, configuration files, and a limited amount of data. Scratch spaces are designed for storing large amounts of data in the time spanned by a job, or a series of jobs. But no filesystem can be used for long-term storage of large amounts of data.

If you need long-term storage, or a dedicated backup storage space, for large amounts of data, solutions can be found within your university:

UCL: contact the CISM team [http://www.uclouvain.be/73254.html]. Access to the storage facility requires a distinct CISM account and is billable.

ULB: TBA

UMons: no dedicated solution exists; you’ll need to resort to departmental resources.

UNamur: contact Frédéric Wautelet. Access to the storage facility is restricted to UNamur and is billed by the year.

ULg: no dedicated solution exists; you’ll need to resort to departmental resources.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Using pre-installed software

Software that is most used on clusters is already pre-installed on the CÉCI clusters: Fortran and C/C++ compilers, popular interpreters like Python or R, numerical libraries such as BLAS and LAPACK, and message passing libraries (e.g. OpenMPI). In addition, several versions of these software are often installed.

If you are already familiar with an environment modules tool and you are only interested in the list of software available in the CÉCI clusters, go to the Software installed in the clusters section. Otherwise, please keep reading to learn how to enable the software you want.

Loading software with the module command

The installed software can be enabled through the use of the module command, its main options are:

module  avail | av            list available software (modules)
module  load | add [module]   set up the environment to use the software
module  list                  list currently loaded software
module  purge                 clears the environment
module  spider                list all possible modules
module  show                  show the commands in the module file
module  help                  get help

For instance, module available openmpi will list all available OpenMPI modules. Assuming it returns something like

OpenMPI/2.1.1-GCC-6.4.0-2.28
OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28
OpenMPI/3.1.1-GCC-7.3.0-2.30

then with the command

module load OpenMPI/2.1.1-GCC-6.4.0-2.28

you will enable OpenMPI version 2.1.1 compiled with GCC version 6.4.0. The naming convention for the available modules is always of the form software/version-toolchain (more on the toolchain part below).

After doing this, when you run e.g. mpicc or mpirun without specifying the full path, you will be running that specific version of OpenMPI compilers or launch script.

Note

The module load command can be abbreviated as ml

Enabling or loading a given module actually sets a group of environment variables such as $LD_LIBRARY_PATH, $CPATH, $PATH, ... to point to the custom installation paths of the software. You can verify how those variables change when loading/unloading a given module. As we provide several versions of the same software and libraries, each one of them must be installed in different custom locations. It’s not possible to rely only on the standard unique location on Unix systems at /usr/lib, /usr/include, /usr/bin, ... used for typical installations with only a single version of each software.

Most of the software requires several dependencies to be built, for instance in the example above we needed at least GCC v6.4.0. Loading the module will also load all the required dependencies at build time. You can verifiy which were the rest of dependencies pulled with

module list

If you want to reset the environment (unload all the modules) use the command

module purge

You will have to do so if you want to load another version of the software, or test to compile your own software with different compiler and libraries versions.

As the module tool is in charge of dynamically modifying your environment variables, it is recommended to not modify the variables $LD_LIBRARY_PATH and $CPATH in your .bashrc, unless you know precisely what you are doing. If you are told to so in the installation instructions of a software you need, please contact us for support, that might save you a lot of time in some cases.

For advanced users, the exact behaviour of the module (setting PATHs, enviroment variables, aliases, etc.) and which are its dependencies can be discovered with the command

module show

For instance:

[you@lemaitre3 ~]$ module show HDF5
------------------------------------------------------------------------------------------------------------------------
   /opt/cecisw/arch/easybuild/2018b/modules/all/HDF5/1.10.2-intel-2018b.lua:
------------------------------------------------------------------------------------------------------------------------
help([[
Description
===========
HDF5 is a data model, library, and file format for storing and managing data.
 It supports an unlimited variety of datatypes, and is designed for flexible
 and efficient I/O and for high volume and complex data.

More information
================
 - Homepage: https://portal.hdfgroup.org/display/support
]])
whatis("Description: HDF5 is a data model, library, and file format for storing and managing data.
 It supports an unlimited variety of datatypes, and is designed for flexible
 and efficient I/O and for high volume and complex data.")
whatis("Homepage: https://portal.hdfgroup.org/display/support")
conflict("HDF5")
load("intel/2018b")
load("zlib/1.2.11-GCCcore-7.3.0")
load("Szip/2.1.1-GCCcore-7.3.0")
prepend_path("CPATH","/opt/cecisw/arch/easybuild/2018b/software/HDF5/1.10.2-intel-2018b/include")
prepend_path("LD_LIBRARY_PATH","/opt/cecisw/arch/easybuild/2018b/software/HDF5/1.10.2-intel-2018b/lib")
prepend_path("LIBRARY_PATH","/opt/cecisw/arch/easybuild/2018b/software/HDF5/1.10.2-intel-2018b/lib")
prepend_path("PATH","/opt/cecisw/arch/easybuild/2018b/software/HDF5/1.10.2-intel-2018b/bin")
prepend_path("PKG_CONFIG_PATH","/opt/cecisw/arch/easybuild/2018b/software/HDF5/1.10.2-intel-2018b/lib/pkgconfig")
setenv("EBROOTHDF5","/opt/cecisw/arch/easybuild/2018b/software/HDF5/1.10.2-intel-2018b")
setenv("EBVERSIONHDF5","1.10.2")
setenv("EBDEVELHDF5","/opt/cecisw/arch/easybuild/2018b/software/HDF5/1.10.2-intel-2018b/easybuild/HDF5-1.10.2-intel-2018b-easybuild-devel")
setenv("HDF5_DIR","/opt/cecisw/arch/easybuild/2018b/software/HDF5/1.10.2-intel-2018b")

the details of the compiling options for a module can be found in the installation log file. That log file can be found, once the module is loaded, at the location

$EBROOT<module name>/easybuild/*log

For instance:

[you@lemaitre3 ~]$ ml HDF5
[you@lemaitre3 ~]$ ls $EBROOTHDF5/easybuild/*log
/opt/cecisw/arch/easybuild/2018b/software/HDF5/1.10.2-intel-2018b/easybuild/easybuild-HDF5-1.10.2-20190430.140559.log
[you@lemaitre3 ~]$ ml FFTW
[you@lemaitre3 ~]$ ls $EBROOTFFTW/easybuild/*log
/opt/cecisw/arch/easybuild/2018b/software/FFTW/3.3.8-gompi-2018b/easybuild/easybuild-FFTW-3.3.8-20181011.025049.log

The full configure command used for the compilation is saved in the log file

[you@lemaitre3 ~]$ grep 'INFO cmd " ./configure' $EBROOTHDF5/easybuild/*log
== 2019-04-30 14:00:55,033 run.py:506 INFO cmd " ./configure --prefix=/opt/cecisw/arch/easybuild/2018b/software/HDF5/1.10.2-intel-2018b --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu  --with-szlib=/opt/cecisw/arch/easybuild/2018b/software/Szip/2.1.1-GCCcore-7.3.0  --with-zlib=/opt/cecisw/arch/easybuild/2018b/software/zlib/1.2.11-GCCcore-7.3.0  --with-pic --with-pthread --enable-shared  --enable-cxx --enable-fortran FC="mpiifort"  --enable-unsupported --enable-parallel " exited with exit code 0 and output:

Toolchains and software organisation

The modules for the available software are organised around the notion of toolchains [http://easybuild.readthedocs.io/en/latest/Common-toolchains.html]. A toolchain is a collection of compiler and libraries that are often used and known to interoperate correctly. For instance, the foss (free open source software) toolchain comprises a bundle of

GCC (including gfortran and g++)

OpenMPI

OpenBLAS

ScaLAPACK

FFTW

Toolchains bundles are further organised into releases designated by a year followed by the letter a or the letter b. Releases corresponding to a specific set of versions for each of the module part of the toolchain. The versions are chosen so as to minimize the risks of bugs and incompatibilities.

For instance release 2017b of the foss toolchain contains:

GCC/6.4.0

OpenBLAS/0.2.20

OpenMPI/2.1.1

ScaLAPACK/2.0.2

FFTW/3.3.6

In most of the CÉCI clusters are also provided some releases of the intel toolchain, which comprises a bundle of

Intel compilers (icc, ifort, icpc)

Intel MPI

MKL

Note

For the pre-installed software we provide, which requires at least the set of tools bundled in a toolchain, it’s followed the convention to add to the module name for it a suffix -toolchain-YYYYx to identify the toolchain used at build time (which becomes a dependency to run this software).

Warning

Mixing toolchains (e.g. foss and intel) and releases (e.g. 2021a and 2022b) is not a good idea as it can introduce linking issues and more generally compatibility issues.

Extensions packages/modules for interpreted languages

Many extensions are installed by default for languages that are pre-installed on the clusters. First load the module corresponding to the version of the language that you want then run the corresponding following command.

For Python, you can get the list with:

$ pip freeze
ansible==2.4.2.0
argcomplete==1.7.0
Babel==0.9.6
backports.ssl-match-hostname==3.4.0.2
bzr==2.5.1
cffi==1.6.0
chardet==2.2.1
[...]

Note

If you work with Python virtual environments, make sure to create the virtual env with --system-site-packages to make the system packages visible from the virtual env.

For R, run this in a R shell:

> ip <- installed.packages(.Library)
> ip[, c(1,3)]
                  Package             Version
abc               "abc"               "2.1"
abc.data          "abc.data"          "1.0"
abind             "abind"             "1.4-5"
acepack           "acepack"           "1.4.1"
adabag            "adabag"            "4.1"
ade4              "ade4"              "1.7-8"
adegenet          "adegenet"          "2.1.0"
adephylo          "adephylo"          "1.1-10"
ADGofTest         "ADGofTest"         "0.3"
akima             "akima"             "0.6-2"
[...]

If you do not find the extension you need, you can try installing it by yourself following these procedures: Installing languages extensions

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Software installed in the clusters

As explained in the previous section Using pre-installed software you must use the module tool to enable a given version of a library, compiler or software you need.

Below you can find the list of modules for the software installed in the clusters organised in two different ways:

Full list of software installed in the clusters

Software installed classified by categories

About some commercial software

Matlab: is not installed on the CÉCI clusters but free alternatives are available (e.g. Octave) and the Matlab Runtime (MCR) [https://nl.mathworks.com/products/compiler/mcr/] is installed on all the clusters. A training session [https://www.ceci-hpc.be/training.html] is dedicated each year to the use of Matlab on the clusters.

Gaussian is installed on most clusters, but its use is restricted and granted on a per case basis. You will need to contact both the system administrator of your university and that of the university hosting the cluster you want to use.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Software installed in the clusters »

Full list of software installed in the clusters

A - B - C - D - E - F - G - H - I - J - K - L - M - N - O - P - Q - R - S - T - U - V - W - X - Y - Z

The naming convention for each module follows the format software/version-toolchain, i.e. the module ABINIT/8.6.3-intel-2018a corresponds to the ABINIT software version 8.6.3 compiled with the intel-2018a toolchain. See here for an explanation about toolchains.

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
ABINIT/8.10.2-intel-2017b |  |  |  | ✓ |
ABINIT/8.10.3-intel-2018b |  | ✓ |  | ✓ |
ABINIT/8.2.2-foss-2016b |  |  |  | ✓ |
ABINIT/8.4.4-intel-2017b |  |  |  | ✓ |
ABINIT/8.6.3-intel-2018a |  |  |  | ✓ |
ABINIT/9.2.2-intel-2020a |  | ✓ |  | ✓ |
ABINIT/9.4.0-intel-2020b |  | ✓ |  |  |
ABySS/2.0.3-foss-2017b |  |  |  |  | ✓
ABySS/2.0.3-foss-2018b |  | ✓ |  |  |
ABySS/2.2.5-foss-2020b |  | ✓ |  |  |
ACTC/1.1-GCCcore-7.3.0 |  | ✓ | ✓ |  | ✓
ALBULA/3.2.0-2-foss-2016b-Python-2.7.12 |  | ✓ |  |  |
AmberTools/17-intel-2019b |  |  |  | ✓ |
AmberTools/20-intel-2019b |  |  |  | ✓ |
Anaconda3/2020.02 |  |  |  |  | ✓
ant/1.10.5-Java-1.8 |  | ✓ | ✓ | ✓ |
ant/1.10.7-Java-11 |  |  | ✓ | ✓ |
ant/1.9.7-Java-1.8.0_92 |  |  |  | ✓ |
ANTLR/2.7.7-GCCcore-7.3.0 |  |  | ✓ | ✓ |
ANTLR/2.7.7-GCCcore-8.3.0 |  |  |  | ✓ |
ANTLR/2.7.7-intel-2017b |  |  | ✓ | ✓ |
AOCC/2.3.0-GCCcore-9.3.0 | ✓ | ✓ |  |  |
AOCC/3.0.0-GCCcore-10.2.0 | ✓ |  |  |  |
archspec/0.1.0-GCCcore-8.3.0-Python-3.7.4 |  | ✓ |  |  |
archspec/0.1.0-GCCcore-9.3.0-Python-3.8.2 | ✓ |  |  |  |
Armadillo/8.400.0-foss-2018a |  | ✓ |  |  |
Armadillo/9.600.5-foss-2018b |  |  | ✓ |  |
arpack-ng/3.5.0-foss-2018a |  | ✓ | ✓ | ✓ |
arpack-ng/3.5.0-foss-2018b |  |  | ✓ | ✓ | ✓
arpack-ng/3.5.0-intel-2017b |  |  | ✓ |  | ✓
arpack-ng/3.7.0-foss-2019b |  | ✓ | ✓ |  | ✓
arpack-ng/3.7.0-intel-2019b |  |  |  | ✓ |
astropy/2.0.14-foss-2019a |  |  |  | ✓ |
astropy/4.0.1-foss-2020a |  |  |  | ✓ |
astropy/4.0.1-foss-2020a-Python-3.8.2 | ✓ |  |  | ✓ |
at-spi2-atk/2.34.1-GCCcore-8.3.0 |  |  | ✓ |  |
at-spi2-core/2.34.0-GCCcore-8.3.0 |  |  | ✓ |  |
ATK/2.20.0-intel-2016a |  |  |  | ✓ |
ATK/2.28.1-foss-2018b |  |  | ✓ |  | ✓
ATK/2.34.1-GCCcore-8.3.0 |  |  | ✓ |  |
AtomPAW/4.1.0.5-intel-2018b |  | ✓ |  | ✓ |
AUGUSTUS/3.3.2-foss-2018b-Python-2.7.15 |  | ✓ |  |  |
AUGUSTUS/3.3.2-intel-2018b-Python-2.7.15 |  |  |  | ✓ |
AutoDock/4.2.6-foss-2016b |  | ✓ |  |  |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
BamTools/2.5.1-foss-2018b |  | ✓ |  |  |
BamTools/2.5.1-GCC-8.3.0 | ✓ | ✓ |  | ✓ |
BamTools/2.5.1-intel-2018b |  |  |  | ✓ |
Bazel/0.11.1-GCCcore-6.4.0 |  |  |  |  | ✓
Bazel/0.18.0-GCCcore-7.3.0 |  |  | ✓ | ✓ | ✓
Bazel/0.20.0-GCCcore-7.3.0 |  |  |  |  | ✓
Bazel/0.26.1-GCCcore-8.3.0 | ✓ |  | ✓ | ✓ | ✓
Bazel/0.29.1-GCCcore-8.3.0 |  |  |  |  | ✓
Bazel/2.0.0-GCCcore-8.3.0 |  |  |  | ✓ |
Bazel/3.4.1-GCCcore-8.3.0 |  |  | ✓ |  | ✓
Bazel/3.7.2-GCCcore-10.2.0 | ✓ |  |  |  |
BCFtools/1.10.2-GCC-9.3.0 | ✓ |  |  |  |
BCFtools/1.11-GCC-10.2.0 | ✓ |  |  |  |
BCFtools/1.9-foss-2018b |  | ✓ |  |  |
BCFtools/1.9-intel-2018b |  |  |  | ✓ |
beagle-lib/3.0.1-foss-2018a |  | ✓ | ✓ |  |
beagle-lib/3.0.2-foss-2018b |  | ✓ | ✓ |  |
beagle-lib/3.0.2-foss-2018b-CUDA-9.2.88 |  |  | ✓ |  | ✓
beagle-lib/3.0.2-fosscuda-2018b |  |  | ✓ |  |
beagle-lib/3.1.2-GCC-8.3.0 |  |  |  |  | ✓
Beast/2.5.0-foss-2018a |  | ✓ |  |  |
Beast/2.5.1-foss-2018b |  |  | ✓ |  |
Beast/2.6.2-foss-2018b |  | ✓ |  |  |
BEDTools/2.29.2-GCC-8.3.0 | ✓ | ✓ |  | ✓ |
BioPerl/1.7.2-foss-2018b-Perl-5.28.0 |  |  |  | ✓ |
BioPerl/1.7.2-GCCcore-8.3.0 |  |  |  | ✓ |
Biopython/1.72-foss-2018b-Python-2.7.15 |  | ✓ |  | ✓ |
BLAST+/2.2.31 |  |  |  | ✓ |
BLAST+/2.7.1-foss-2018b |  | ✓ |  | ✓ |
BLAST+/2.7.1-intel-2018b |  |  |  | ✓ |
BLAST+/2.8.1-foss-2018b |  | ✓ |  | ✓ |
BLAST/2.11.0-Linux_x86_64 | ✓ |  |  |  |
BLAST/2.2.26-Linux_x86_64 |  |  |  | ✓ |
BLAST/2.8.1-Linux_x86_64 |  |  | ✓ |  |
BLAT/3.5-foss-2018b |  | ✓ |  |  |
BLIS/0.8.0-GCC-9.3.0 | ✓ |  |  |  |
BLIS/0.8.0-GCCcore-9.3.0 | ✓ |  |  |  |
BLIS/2.2-GCCcore-9.3.0-amd | ✓ |  |  |  |
Blosc/1.17.1-GCCcore-8.3.0 |  | ✓ |  |  |
Boost.Python/1.66.0-foss-2018a-Python-3.6.4 |  |  | ✓ |  |
Boost.Python/1.66.0-intel-2018a-Python-3.6.4 |  |  | ✓ |  |
Boost.Python/1.67.0-foss-2018b-Python-2.7.15 |  | ✓ |  | ✓ | ✓
Boost.Python/1.67.0-foss-2018b-Python-3.6.6 |  |  | ✓ |  |
Boost.Python/1.67.0-fosscuda-2018b-Python-3.6.6 |  |  | ✓ |  |
Boost.Python/1.67.0-intel-2018b-Python-2.7.15 |  |  |  | ✓ |
Boost.Python/1.67.0-intel-2018b-Python-3.6.6 |  |  | ✓ |  |
Boost.Python/1.74.0-GCC-10.2.0 | ✓ |  |  |  |
Boost/1.60.0-foss-2018a |  |  |  | ✓ |
Boost/1.61.0-foss-2016b |  |  |  | ✓ |
Boost/1.61.0-intel-2016a-Python-2.7.11 |  |  |  | ✓ |
Boost/1.63.0-foss-2016b-Python-3.5.2 |  |  |  | ✓ |
Boost/1.65.1-foss-2017b |  |  |  | ✓ | ✓
Boost/1.65.1-foss-2017b-Python-2.7.14 |  |  |  |  | ✓
Boost/1.65.1-foss-2017b-Python-3.6.3 |  |  |  |  | ✓
Boost/1.65.1-foss-2018a |  |  |  | ✓ |
Boost/1.65.1-intel-2017b-Python-3.6.3 |  |  |  |  | ✓
Boost/1.66.0-foss-2018a |  | ✓ | ✓ | ✓ |
Boost/1.66.0-intel-2017b |  |  | ✓ | ✓ |
Boost/1.66.0-intel-2018a |  |  | ✓ |  |
Boost/1.66.0-intel-2018a-Python-3.6.4 |  |  |  | ✓ |
Boost/1.67.0-foss-2018b |  | ✓ | ✓ | ✓ | ✓
Boost/1.67.0-fosscuda-2018b |  |  | ✓ |  |
Boost/1.67.0-intel-2018b |  | ✓ | ✓ | ✓ |
Boost/1.68.0-foss-2018b-Python-2.7.15 |  | ✓ |  |  | ✓
Boost/1.68.0-foss-2018b-Python-3.6.6 |  | ✓ |  |  |
Boost/1.68.0-intel-2018b-Python-2.7.15 |  | ✓ |  |  |
Boost/1.68.0-intel-2018b-Python-3.6.6 |  | ✓ |  |  | ✓
Boost/1.71.0-gompi-2019b | ✓ | ✓ | ✓ | ✓ | ✓
Boost/1.71.0-iimpi-2019b |  |  |  | ✓ |
Boost/1.72.0-gompi-2020a | ✓ | ✓ |  | ✓ |
Boost/1.72.0-iimpi-2020a |  |  |  | ✓ |
Boost/1.74.0-GCC-10.2.0 | ✓ | ✓ | ✓ | ✓ | ✓
Bottleneck/1.3.2-foss-2020a-Python-3.8.2 |  | ✓ |  |  |
Bowtie/1.2.2-foss-2018b |  | ✓ |  | ✓ |
Bowtie2/2.3.4.2-foss-2018b |  | ✓ | ✓ | ✓ | ✓
Bowtie2/2.3.4.2-intel-2018b |  |  |  |  | ✓
Bowtie2/2.3.5.1-iccifort-2019.5.281 |  |  |  |  | ✓
BUSCO/3.0.2-foss-2018b-Python-2.7.15 |  | ✓ |  |  |
BUSCO/3.0.2-intel-2018b-Python-2.7.15 |  |  |  | ✓ |
BWA/0.7.17-foss-2018b |  | ✓ |  |  |
BWA/0.7.17-foss-2019b |  |  |  | ✓ |
BWA/0.7.17-GCC-9.3.0 | ✓ |  |  |  |
bwidget/1.9.14-GCCcore-8.3.0 |  |  | ✓ |  | ✓

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
canu/1.9-GCCcore-8.3.0-Java-11 |  | ✓ |  | ✓ |
CD-HIT/4.6.8-foss-2018b |  | ✓ |  | ✓ |
CDO/1.7.2-intel-2016a |  |  |  | ✓ |
CDO/1.8.2-intel-2018a |  |  |  | ✓ |
CDO/1.9.10-foss-2020b | ✓ |  |  |  |
CDO/1.9.2-intel-2017b |  |  |  | ✓ |
CDO/1.9.8-intel-2019b |  |  |  | ✓ |
CFITSIO/3.47-foss-2020a |  |  |  | ✓ |
CFITSIO/3.47-GCCcore-8.2.0 |  |  |  | ✓ |
CFITSIO/3.47-GCCcore-8.3.0 | ✓ |  | ✓ | ✓ |
CFITSIO/3.48-GCCcore-9.3.0 | ✓ |  |  |  |
CFITSIO/3.49-GCCcore-10.2.0 | ✓ |  |  |  |
cftime/1.0.0-intel-2018a-Python-3.6.4 |  |  |  | ✓ |
CGAL/4.11-foss-2017b-Python-2.7.14 |  |  |  | ✓ |
CGAL/4.11.1-foss-2018b-Python-2.7.15 |  |  | ✓ | ✓ | ✓
CGAL/4.14.1-foss-2019b-Python-3.7.4 |  |  | ✓ |  | ✓
CGAL/4.14.1-intel-2019b-Python-3.7.4 |  |  |  | ✓ |
CGAL/4.14.3-gompi-2020a-Python-3.8.2 | ✓ |  |  | ✓ |
CGAL/4.8.1-foss-2016b |  |  |  | ✓ |
CGAL/5.2-gompi-2020b | ✓ |  |  |  |
Check/0.15.2-GCCcore-9.3.0 |  | ✓ |  |  |
Clang/11.0.0-GCCcore-9.3.0 | ✓ |  |  |  |
Clang/6.0.1-GCC-6.4.0-2.28 |  |  | ✓ |  |
Clang/7.0.0-GCC-6.4.0-2.28 |  |  | ✓ |  |
CLHEP/2.4.0.0-intel-2017b |  |  | ✓ |  |
CLHEP/2.4.1.3-foss-2019b |  |  | ✓ | ✓ |
CLHEP/2.4.4.0-GCC-10.2.0 | ✓ |  |  |  |
CMake/3.10.0-GCCcore-6.4.0 |  |  | ✓ | ✓ |
CMake/3.10.1-GCCcore-6.4.0 |  |  | ✓ | ✓ |
CMake/3.10.2-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ |
CMake/3.11.1-GCCcore-6.4.0 |  |  |  |  | ✓
CMake/3.11.4-GCCcore-6.4.0 |  |  | ✓ |  |
CMake/3.11.4-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
CMake/3.12.1-GCCcore-6.4.0 |  |  |  | ✓ |
CMake/3.12.1-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
CMake/3.13.3-GCCcore-8.2.0 |  |  |  | ✓ |
CMake/3.15.3-GCCcore-8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
CMake/3.16.4-GCCcore-9.3.0 | ✓ | ✓ |  | ✓ |
CMake/3.18.4-GCCcore-10.2.0 | ✓ | ✓ | ✓ | ✓ | ✓
CMake/3.4.3-foss-2016b |  |  |  | ✓ |
CMake/3.4.3-foss-2018a |  |  |  | ✓ |
CMake/3.4.3-intel-2016a |  |  |  | ✓ |
CMake/3.5.1-intel-2016a |  |  |  | ✓ |
CMake/3.5.2-foss-2016b |  |  |  | ✓ |
CMake/3.5.2-intel-2016a |  |  |  | ✓ |
CMake/3.6.1-foss-2016b |  |  |  | ✓ |
CMake/3.7.1-foss-2016b |  |  |  | ✓ |
CMake/3.7.2-foss-2016b |  |  |  | ✓ |
CMake/3.7.2-foss-2018a |  |  |  | ✓ |
CMake/3.9.1-GCCcore-6.4.0 |  |  | ✓ | ✓ | ✓
CMake/3.9.4-GCCcore-6.4.0 |  |  | ✓ | ✓ | ✓
CMake/3.9.5-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ | ✓
CMake/3.9.6 |  |  | ✓ | ✓ |
CNT-ILP/20171031-GCC-8.2.0-2.31.1 |  |  |  | ✓ |
CoLoRMap/20180306-foss-2018b |  | ✓ |  |  |
configurable-http-proxy/4.2.1-GCCcore-10.2.0 | ✓ |  |  |  |
CP2K/5.1-intel-2017b |  | ✓ |  | ✓ |
CP2K/6.1-intel-2018a |  |  | ✓ |  |
CP2K/6.1-intel-2018b |  |  |  |  | ✓
CP2K/7.1-intel-2020a |  |  |  | ✓ |
CPLEX/12.9-GCCcore-8.2.0 |  |  |  | ✓ |
CRYSTAL/17.1.0.2-iomkl-2015b-ifort14-P |  | ✓ |  |  |
CRYSTAL/17.1.0.2-iomkl-2017b-ifort17-P |  | ✓ |  |  |
CRYSTAL/17.1.0.2-iomkl-2017b-P |  | ✓ |  |  |
CubeGUI/4.4.4-GCCcore-9.3.0 | ✓ |  |  |  |
CubeLib/4.4.4-GCCcore-9.3.0 | ✓ |  |  |  |
CubeWriter/4.4.3-GCCcore-9.3.0 | ✓ |  |  |  |
CUDA/10.0.130 |  |  | ✓ |  |
CUDA/10.1.243-GCC-8.3.0 |  | ✓ | ✓ |  | ✓
CUDA/11.0.2-GCC-9.3.0 |  | ✓ |  |  |
CUDA/7.5.18-iccifort-2016.3.210-GCC-4.9.3-2.25 |  |  |  |  | ✓
CUDA/8.0.44-GCC-5.4.0-2.26 |  |  |  |  | ✓
CUDA/8.0.44-iccifort-2016.3.210-GCC-5.4.0-2.26 |  |  |  |  | ✓
CUDA/8.0.61_375.26-GCC-5.4.0-2.26 |  |  | ✓ |  | ✓
CUDA/9.0.176-GCC-6.4.0-2.28 |  |  | ✓ |  | ✓
CUDA/9.0.176-iccifort-2017.4.196-GCC-6.4.0-2.28 |  |  |  |  | ✓
CUDA/9.1.85 |  |  | ✓ |  | ✓
CUDA/9.1.85-GCC-6.4.0-2.28 |  |  | ✓ | ✓ | ✓
CUDA/9.2.88 |  |  | ✓ |  | ✓
CUDA/9.2.88-GCC-6.4.0-2.28 |  |  | ✓ |  |
CUDA/9.2.88-GCC-7.3.0-2.30 |  | ✓ | ✓ |  | ✓
CUDAcore/11.0.2 |  | ✓ |  |  |
cuDNN/7.0.5.15-fosscuda-2017b |  |  |  |  | ✓
cuDNN/7.1.4.18-fosscuda-2018b |  |  | ✓ |  | ✓
cuDNN/7.6.4.38-gcccuda-2019b |  |  | ✓ |  | ✓
cutadapt/2.8-GCCcore-8.3.0-Python-3.7.4 | ✓ |  |  | ✓ |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
DB/18.1.32-GCCcore-8.3.0 |  | ✓ |  | ✓ | ✓
DB/18.1.32-GCCcore-9.3.0 |  | ✓ |  |  |
DB/18.1.40-GCCcore-10.2.0 |  | ✓ | ✓ | ✓ | ✓
DBG2OLC/20200724-GCC-8.3.0 |  | ✓ |  |  |
DDEELS/3.0_alpha-intel-2019b |  | ✓ |  |  |
DFT-D3/3.2.0-intel-2019b |  |  |  | ✓ |
DFTB+/17.1-intel-2017b-Python-2.7.14 |  |  | ✓ |  |
DFTB+/19.1-foss-2019b-Python-2.7.16 |  | ✓ |  |  | ✓
DFTB+/19.1-foss-2019b-Python-2.7.16-mpi |  |  |  | ✓ |
DFTB+/19.1-intel-2019b-Python-2.7.16 |  |  |  | ✓ |
dftd3-lib/0.9-GCC-8.3.0 |  | ✓ |  | ✓ | ✓
DMTCP/2.5.0-foss-2018a |  |  |  | ✓ |
DMTCP/2.5.2-GCCcore-8.3.0 |  |  |  | ✓ |
double-conversion/3.1.4-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
double-conversion/3.1.5-GCCcore-10.2.0 | ✓ |  |  |  |
double-conversion/3.1.5-GCCcore-9.3.0 | ✓ |  |  | ✓ |
Doxygen/1.8.11-foss-2016b |  |  |  | ✓ |
Doxygen/1.8.11-foss-2018a |  |  |  | ✓ |
Doxygen/1.8.11-intel-2016a |  |  |  | ✓ |
Doxygen/1.8.13-GCCcore-6.4.0 |  |  | ✓ | ✓ | ✓
Doxygen/1.8.14-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
Doxygen/1.8.16-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
Doxygen/1.8.17-GCCcore-9.3.0 |  | ✓ |  | ✓ |
Doxygen/1.8.20-GCCcore-10.2.0 |  | ✓ | ✓ |  | ✓

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
ecCodes/2.15.0-iimpi-2019b |  |  |  | ✓ |
ecCodes/2.20.0-gompi-2020b | ✓ |  |  |  |
ecCodes/2.9.2-intel-2018b |  |  | ✓ |  |
Eigen/3.2.9-foss-2016b |  |  |  | ✓ |
Eigen/3.3.4 |  |  | ✓ | ✓ | ✓
Eigen/3.3.5 |  |  |  |  | ✓
Eigen/3.3.7 |  | ✓ | ✓ | ✓ | ✓
Eigen/3.3.7-GCCcore-9.3.0 | ✓ | ✓ |  | ✓ |
Eigen/3.3.8-GCCcore-10.2.0 | ✓ |  | ✓ |  | ✓
ELPA/2016.11.001.pre-foss-2018b |  |  |  | ✓ |
ELPA/2016.11.001.pre-intel-2018b |  |  | ✓ | ✓ | ✓
ELPA/2018.05.001-foss-2018b |  |  | ✓ |  |
ELPA/2019.11.001-foss-2019b | ✓ |  |  | ✓ |
ELPA/2019.11.001-intel-2019b |  | ✓ | ✓ | ✓ | ✓
ELPA/2020.11.001-foss-2020b | ✓ |  |  |  |
ELSI/2.5.0-intel-2019b-PEXSI |  |  |  | ✓ |
EMBOSS/6.6.0-foss-2018b |  | ✓ |  |  |
EMBOSS/6.6.0-intel-2018b |  |  |  | ✓ |
ESMF/7.0.2-intel-2017b |  |  | ✓ |  |
ETSF_IO/1.0.4-intel-2018b |  |  | ✓ | ✓ |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
FASTA/36.3.8-foss-2019b |  | ✓ |  |  |
FASTA/36.3.8h_04-May-2020-foss-2019b |  | ✓ |  |  |
FastQC/0.11.8-Java-1.8 |  |  |  | ✓ |
FATSLiM/0.2.1-foss-2018a-Python-3.6.4 |  | ✓ |  |  |
fatslim/0.2.1-foss-2018a-Python-3.6.4 |  |  | ✓ |  |
FFmpeg/3.0.2-intel-2016a |  |  |  | ✓ |
FFmpeg/4.1-foss-2018b |  |  | ✓ |  |
FFmpeg/4.1-intel-2018b |  | ✓ | ✓ |  | ✓
FFmpeg/4.2.1-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
FFmpeg/4.2.2-GCCcore-9.3.0 | ✓ |  |  | ✓ |
FFmpeg/4.3.1-GCCcore-10.2.0 | ✓ |  |  |  |
FFTW/3.3.4-gompi-2016b |  |  |  | ✓ | ✓
FFTW/3.3.6-gompi-2017b |  | ✓ | ✓ | ✓ | ✓
FFTW/3.3.6-gompic-2017b |  |  |  |  | ✓
FFTW/3.3.6-intel-2017b |  | ✓ | ✓ | ✓ |
FFTW/3.3.7-gompi-2018a |  | ✓ | ✓ | ✓ |
FFTW/3.3.7-gompic-2018a |  |  | ✓ |  |
FFTW/3.3.7-intel-2018a |  |  | ✓ |  |
FFTW/3.3.8-gompi-2018b |  | ✓ | ✓ | ✓ | ✓
FFTW/3.3.8-gompi-2019a |  |  |  | ✓ |
FFTW/3.3.8-gompi-2019b | ✓ | ✓ | ✓ | ✓ | ✓
FFTW/3.3.8-gompi-2020a | ✓ | ✓ |  | ✓ |
FFTW/3.3.8-gompi-2020b | ✓ | ✓ | ✓ | ✓ | ✓
FFTW/3.3.8-gompic-2018b |  | ✓ | ✓ |  | ✓
FFTW/3.3.8-gompic-2019b |  | ✓ | ✓ |  | ✓
FFTW/3.3.8-gompic-2020a |  | ✓ |  |  |
FFTW/3.3.8-intel-2018b |  |  |  |  | ✓
FFTW/3.3.8-intel-2019b |  |  | ✓ |  | ✓
FFTW/3.3.8-intel-2020a |  |  |  | ✓ |
FFTW/3.3.8-intel-2020b | ✓ |  |  |  |
FIAT/1.6.0-intel-2016a-Python-2.7.11 |  |  |  | ✓ |
Fiona/1.8.16-foss-2020a-Python-3.8.2 | ✓ |  |  |  |
FLAC/1.3.3-GCCcore-10.2.0 |  |  | ✓ |  | ✓
flatbuffers-python/1.12-GCCcore-10.2.0 | ✓ |  |  |  |
flatbuffers/1.12.0-GCCcore-10.2.0 | ✓ |  |  |  |
flatbuffers/1.12.0-GCCcore-8.3.0 |  |  | ✓ |  | ✓
FLTK/1.3.4-foss-2017b |  |  | ✓ |  |
FLTK/1.3.4-foss-2018a |  | ✓ | ✓ | ✓ |
FLTK/1.3.4-foss-2018b |  |  | ✓ | ✓ | ✓
FLTK/1.3.4-fosscuda-2018b |  |  | ✓ |  |
FLTK/1.3.4-intel-2017b |  |  |  |  | ✓
FLTK/1.3.4-intel-2018b |  | ✓ | ✓ |  | ✓
FLTK/1.3.5-GCC-8.3.0 |  |  | ✓ |  | ✓
FLUENT/14.0 |  |  |  | ✓ |
FLUENT/18.2 |  |  |  | ✓ |
Flye/2.6-foss-2019b-Python-3.7.4 |  | ✓ |  |  |
Flye/2.6-intel-2019b-Python-3.7.4 |  |  |  | ✓ |
foss/2016b |  |  |  | ✓ | ✓
foss/2017b |  | ✓ | ✓ | ✓ | ✓
foss/2018a |  | ✓ | ✓ | ✓ |
foss/2018b |  | ✓ | ✓ | ✓ | ✓
foss/2019a |  |  |  | ✓ |
foss/2019b | ✓ | ✓ | ✓ | ✓ | ✓
foss/2020a | ✓ | ✓ |  | ✓ |
foss/2020b | ✓ | ✓ | ✓ | ✓ | ✓
fosscuda/2017b |  |  |  |  | ✓
fosscuda/2018a |  |  | ✓ |  |
fosscuda/2018b |  | ✓ | ✓ |  | ✓
fosscuda/2019b |  | ✓ | ✓ |  | ✓
fosscuda/2020a |  | ✓ |  |  |
FreeXL/1.0.2-foss-2016b |  |  |  | ✓ |
FriBidi/1.0.1-foss-2018a |  |  |  | ✓ |
FriBidi/1.0.10-GCCcore-10.2.0 | ✓ |  |  |  |
FriBidi/1.0.2-GCCcore-6.4.0 |  |  | ✓ |  |
FriBidi/1.0.5-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
FriBidi/1.0.5-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
FriBidi/1.0.9-GCCcore-9.3.0 | ✓ |  |  | ✓ |
FSL/6.0.3-foss-2019b-Python-3.7.4 |  | ✓ |  | ✓ |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
g2clib/1.6.0-intel-2017b |  |  | ✓ |  |
g2lib/1.4.0-intel-2017b |  |  | ✓ |  |
g2molden/0.1-intel-2017b |  | ✓ |  |  |
g2molden/0.2-intel-2017b |  | ✓ |  |  |
GATE/9.0-foss-2019b-Python-3.7.4 |  |  |  | ✓ |
GATK/4.1.4.1-GCCcore-8.3.0-Java-11 | ✓ |  |  | ✓ |
GATK/4.2.0.0-GCCcore-10.2.0-Java-1.8 |  | ✓ |  |  |
Gaussian/09_D01-PGI-12.10 |  | ✓ |  |  |
Gaussian/16_A03-PGI-16.5 |  | ✓ |  |  |
Gaussian/16_A03-PGI-16.5-betaprint-2 |  | ✓ |  |  |
gc/7.6.0-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ |
gc/7.6.12-GCCcore-8.3.0 |  | ✓ |  |  |
gc/7.6.12-GCCcore-9.3.0 |  |  |  | ✓ |
gc/7.6.4-GCCcore-7.3.0 |  |  |  |  | ✓
GCC/10.2.0 | ✓ | ✓ | ✓ | ✓ | ✓
GCC/4.9.3-2.25 |  |  |  | ✓ |
GCC/5.4.0-2.26 |  |  | ✓ | ✓ | ✓
GCC/6.3.0-2.27 |  |  | ✓ | ✓ |
GCC/6.4.0-2.28 |  | ✓ | ✓ | ✓ | ✓
GCC/7.3.0-2.30 |  | ✓ | ✓ | ✓ | ✓
GCC/8.2.0-2.31.1 |  |  | ✓ | ✓ |
GCC/8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
GCC/9.2.0-2.32 |  | ✓ |  |  |
GCC/9.3.0 | ✓ | ✓ |  | ✓ |
GCCcore/10.2.0 |  | ✓ | ✓ | ✓ | ✓
GCCcore/4.9.3 |  |  |  | ✓ | ✓
GCCcore/5.4.0 |  |  | ✓ | ✓ | ✓
GCCcore/6.3.0 |  |  | ✓ | ✓ |
GCCcore/6.4.0 |  |  | ✓ | ✓ | ✓
GCCcore/7.3.0 |  |  | ✓ | ✓ | ✓
GCCcore/8.2.0 |  |  | ✓ | ✓ |
GCCcore/8.3.0 |  |  | ✓ | ✓ | ✓
GCCcore/9.3.0 |  |  |  | ✓ |
gcccuda/2016.10 |  |  |  |  | ✓
gcccuda/2017b |  |  |  |  | ✓
gcccuda/2018a |  |  | ✓ |  |
gcccuda/2018b |  | ✓ | ✓ |  | ✓
gcccuda/2019b |  | ✓ | ✓ |  | ✓
gcccuda/2020a |  | ✓ |  |  |
GDAL/2.1.0-foss-2016b |  |  |  | ✓ |
GDAL/2.1.3-foss-2016b-Python-2.7.12 |  |  |  | ✓ |
GDAL/2.2.2-intel-2017b-Python-2.7.14-HDF5-1.8.19 |  |  | ✓ |  |
GDAL/2.2.3-foss-2017b-Python-2.7.14 |  |  |  | ✓ | ✓
GDAL/2.2.3-foss-2017b-Python-3.6.3 |  |  |  | ✓ |
GDAL/2.2.3-foss-2018a-Python-3.6.3 |  |  |  | ✓ |
GDAL/2.2.3-foss-2018b-Python-3.6.6 |  | ✓ | ✓ | ✓ | ✓
GDAL/2.2.3-intel-2018b-Python-3.6.6 |  |  |  |  | ✓
GDAL/3.0.2-foss-2019b-Python-3.7.4 | ✓ |  |  |  |
GDAL/3.0.2-fosscuda-2019b-Python-3.7.4 |  |  | ✓ |  |
GDAL/3.0.4-foss-2020a-Python-3.8.2 | ✓ | ✓ |  | ✓ |
GDAL/3.2.1-foss-2020b |  |  | ✓ |  | ✓
GDRCopy/2.1-GCCcore-9.3.0-CUDA-11.0.2 |  | ✓ |  |  |
Geant4/10.04-intel-2017b |  |  | ✓ |  |
Geant4/10.6-foss-2019b |  |  | ✓ | ✓ |
Geant4/10.7.1-GCC-10.2.0 | ✓ |  |  |  |
geopandas/0.8.1-foss-2020a-Python-3.8.2 | ✓ |  |  |  |
GEOS/3.5.0-intel-2016a-Python-2.7.11 |  |  |  | ✓ |
GEOS/3.6.1-foss-2016b-Python-2.7.12 |  |  |  | ✓ |
GEOS/3.6.2-foss-2017b-Python-2.7.14 |  |  |  | ✓ | ✓
GEOS/3.6.2-foss-2017b-Python-3.6.3 |  |  |  | ✓ |
GEOS/3.6.2-foss-2018a-Python-3.6.3 |  |  |  | ✓ |
GEOS/3.6.2-foss-2018b-Python-3.6.6 |  | ✓ | ✓ | ✓ | ✓
GEOS/3.6.2-intel-2017b-Python-2.7.14 |  |  | ✓ |  |
GEOS/3.6.2-intel-2017b-Python-3.6.3 |  |  |  | ✓ |
GEOS/3.6.2-intel-2018b-Python-3.6.6 |  |  |  |  | ✓
GEOS/3.8.0-GCC-8.3.0-Python-3.7.4 | ✓ |  | ✓ |  |
GEOS/3.8.1-GCC-9.3.0-Python-3.8.2 | ✓ | ✓ |  | ✓ |
GEOS/3.9.1-GCC-10.2.0 |  |  | ✓ |  | ✓
gflags/2.2.1-intel-2017b |  |  |  | ✓ |
gflags/2.2.2-GCCcore-8.3.0 |  |  | ✓ | ✓ |
giflib/5.2.1-GCCcore-10.2.0 | ✓ |  |  |  |
giflib/5.2.1-GCCcore-8.3.0 |  |  | ✓ |  | ✓
git/2.13.1-intel-2018a |  |  |  | ✓ |
git/2.23.0-GCCcore-8.3.0-nodocs | ✓ | ✓ | ✓ | ✓ | ✓
git/2.28.0-GCCcore-10.2.0-nodocs | ✓ |  |  |  |
Glade/3.8.6-foss-2018b-Python-2.7.15 |  |  |  |  | ✓
glew/2.1.0-GCCcore-8.3.0 |  |  | ✓ | ✓ |
glew/2.2.0-GCCcore-10.2.0-glx | ✓ |  |  |  |
GLibmm/2.49.7-GCCcore-8.3.0 |  | ✓ |  | ✓ |
GlobalArrays/5.7.2-intel-2019b-peigs |  |  | ✓ |  | ✓
glog/0.4.0-GCCcore-8.3.0 |  |  | ✓ | ✓ |
GLPK/4.65-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
GLPK/4.65-GCCcore-6.4.0 |  |  | ✓ | ✓ |
GLPK/4.65-GCCcore-7.3.0 |  |  | ✓ | ✓ |
GLPK/4.65-GCCcore-8.3.0 |  |  | ✓ |  | ✓
GLPK/4.65-GCCcore-9.3.0 |  | ✓ |  | ✓ |
GMAP-GSNAP/2019-03-15-foss-2018b |  | ✓ |  | ✓ |
GMP/6.1.0-intel-2016a |  |  |  | ✓ |
GMP/6.1.1-foss-2016b |  |  |  | ✓ |
GMP/6.1.2-foss-2016b |  |  |  | ✓ |
GMP/6.1.2-foss-2018a |  |  |  | ✓ |
GMP/6.1.2-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ | ✓
GMP/6.1.2-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
GMP/6.1.2-GCCcore-8.2.0 |  |  |  | ✓ |
GMP/6.1.2-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
GMP/6.2.0-GCCcore-10.2.0 |  | ✓ | ✓ |  | ✓
GMP/6.2.0-GCCcore-9.3.0 |  | ✓ |  | ✓ |
gnuplot/5.2.2-foss-2018a |  |  | ✓ | ✓ |
gnuplot/5.2.5-foss-2018b |  |  | ✓ | ✓ | ✓
gnuplot/5.2.6-foss-2018b |  | ✓ |  |  |
gnuplot/5.2.8-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
gnuplot/5.2.8-GCCcore-9.3.0 | ✓ |  |  |  |
gnuplot/5.4.1-GCCcore-10.2.0 | ✓ |  |  |  |
gobff/2020.11 | ✓ |  |  |  |
gompi/2016b |  |  |  | ✓ | ✓
gompi/2017b |  | ✓ | ✓ | ✓ | ✓
gompi/2018a |  | ✓ | ✓ | ✓ |
gompi/2018b |  | ✓ | ✓ | ✓ | ✓
gompi/2019a |  |  |  | ✓ |
gompi/2019b | ✓ | ✓ | ✓ | ✓ | ✓
gompi/2020a | ✓ | ✓ |  | ✓ |
gompi/2020b | ✓ | ✓ | ✓ | ✓ | ✓
gompic/2017b |  |  |  |  | ✓
gompic/2018a |  |  | ✓ |  |
gompic/2018b |  | ✓ | ✓ |  | ✓
gompic/2019b |  | ✓ | ✓ |  | ✓
gompic/2020a |  | ✓ |  |  |
googletest/1.10.0-GCCcore-8.3.0 |  |  |  | ✓ |
Grace/5.1.25-foss-2017b-5build1 |  |  |  | ✓ |
Grace/5.1.25-foss-2019b-5build1 |  | ✓ |  |  |
Grace/5.1.25-intel-2017b-5build1-HDF5-1.8.19 |  |  |  | ✓ |
GraphicsMagick/1.3.28-foss-2018a |  |  | ✓ | ✓ |
GraphicsMagick/1.3.31-foss-2018b |  |  | ✓ | ✓ |
GraphicsMagick/1.3.34-foss-2019b |  |  | ✓ |  | ✓
grib_api/1.16.0-intel-2016a |  |  |  | ✓ |
grib_api/1.16.0-intel-2018a |  |  |  | ✓ |
grib_api/1.24.0-intel-2017b |  |  |  | ✓ |
groff/1.22.4-GCCcore-9.3.0 | ✓ | ✓ |  |  |
GROMACS/2016-foss-2016b-mt |  |  |  | ✓ |
GROMACS/2016.5-intel-2018a |  |  | ✓ |  |
GROMACS/2018.3-foss-2018b |  |  |  | ✓ |
GROMACS/2018.6-fosscuda-2018b |  |  | ✓ |  |
GROMACS/2019-fosscuda-2018b |  | ✓ | ✓ |  |
GROMACS/2020-foss-2019b | ✓ | ✓ |  |  | ✓
GROMACS/2020.3-fosscuda-2019b |  |  |  |  | ✓
GROMACS/2020.4-foss-2020a-Python-3.8.2 | ✓ |  |  |  |
GROMACS/2020.5-fosscuda-2020a-Python-3.8.2 |  | ✓ |  |  |
GROMACS/2020.5-fosscuda-2020a-Python-3.8.2-AVX2_256 |  | ✓ |  |  |
GSL/2.3-foss-2016b |  |  |  | ✓ |
GSL/2.4-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ |
GSL/2.5-GCC-7.3.0-2.30 |  | ✓ | ✓ | ✓ | ✓
GSL/2.5-iccifort-2018.3.222-GCC-7.3.0-2.30 |  |  | ✓ | ✓ | ✓
GSL/2.6-GCC-10.2.0 | ✓ | ✓ | ✓ |  | ✓
GSL/2.6-GCC-8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
GSL/2.6-GCC-9.3.0 | ✓ | ✓ |  | ✓ |
GSL/2.6-iccifort-2019.5.281 |  |  |  | ✓ |
GSL/2.6-iccifort-2020.1.217 |  |  |  | ✓ |
Guile/1.8.8-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ |
Guile/1.8.8-GCCcore-7.3.0 |  |  |  |  | ✓
Guile/1.8.8-GCCcore-8.3.0 |  | ✓ |  |  |
Guile/1.8.8-GCCcore-9.3.0 |  |  |  | ✓ |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
h5py/2.10.0-foss-2019b-Python-3.7.4 | ✓ | ✓ | ✓ | ✓ |
h5py/2.10.0-foss-2020a-Python-3.8.2 |  | ✓ |  | ✓ |
h5py/2.10.0-fosscuda-2019b-Python-3.7.4 |  |  | ✓ |  | ✓
h5py/2.10.0-intel-2019b-Python-3.7.4 |  |  |  | ✓ |
h5py/2.7.1-intel-2017b-Python-2.7.14 |  | ✓ |  |  |
h5py/2.8.0-foss-2018b-Python-3.6.6 |  |  | ✓ | ✓ | ✓
h5py/2.8.0-fosscuda-2018b-Python-3.6.6 |  |  | ✓ |  | ✓
h5py/3.1.0-foss-2020b | ✓ |  |  |  |
HarfBuzz/1.7.5-foss-2018a |  |  | ✓ | ✓ |
HarfBuzz/2.2.0-foss-2018b |  | ✓ | ✓ | ✓ | ✓
HarfBuzz/2.6.4-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
haslr/20200423-GCCcore-8.3.0-Python-3.7.4 |  | ✓ |  |  |
HDF/4.2.13-GCCcore-6.4.0 |  |  | ✓ |  |
HDF/4.2.15-GCCcore-10.2.0 |  |  | ✓ |  | ✓
HDF/4.2.15-GCCcore-9.3.0 |  | ✓ |  |  |
HDF5/1.10.0-patch1-foss-2016b |  |  |  | ✓ |
HDF5/1.10.1-foss-2017b |  |  |  | ✓ | ✓
HDF5/1.10.1-foss-2018a |  | ✓ | ✓ | ✓ |
HDF5/1.10.1-intel-2017b |  | ✓ | ✓ | ✓ |
HDF5/1.10.1-intel-2018a |  |  | ✓ | ✓ |
HDF5/1.10.2-foss-2018b |  | ✓ | ✓ | ✓ | ✓
HDF5/1.10.2-fosscuda-2018b |  |  | ✓ |  | ✓
HDF5/1.10.2-intel-2018b |  | ✓ | ✓ | ✓ | ✓
HDF5/1.10.2-iomkl-2018b |  |  |  | ✓ |
HDF5/1.10.5-gompi-2019b | ✓ | ✓ | ✓ | ✓ | ✓
HDF5/1.10.5-gompic-2019b |  |  | ✓ |  | ✓
HDF5/1.10.5-iimpi-2019b |  |  | ✓ | ✓ | ✓
HDF5/1.10.6-gompi-2020a | ✓ | ✓ |  | ✓ |
HDF5/1.10.6-iimpi-2020a |  | ✓ |  | ✓ |
HDF5/1.10.7-gompi-2020b | ✓ | ✓ | ✓ |  | ✓
HDF5/1.10.7-iimpi-2020b | ✓ | ✓ |  |  |
HDF5/1.8.16-foss-2018a |  |  |  | ✓ |
HDF5/1.8.16-intel-2016a |  |  |  | ✓ |
HDF5/1.8.16-intel-2016a-serial |  |  |  | ✓ |
HDF5/1.8.17-foss-2016b |  |  |  | ✓ |
HDF5/1.8.17-intel-2016a |  |  |  | ✓ |
HDF5/1.8.18-intel-2016a |  |  |  | ✓ |
HDF5/1.8.18-intel-2018a |  |  |  | ✓ |
HDF5/1.8.19-intel-2017b |  |  | ✓ | ✓ |
HEALPix/3.70-GCCcore-10.2.0 | ✓ |  |  |  |
HH-suite/3.2.0-foss-2019b |  |  |  | ✓ |
HISAT2/2.1.0-foss-2018b |  |  |  | ✓ |
HMMER/2.3.2-foss-2018b |  | ✓ |  |  |
HMMER/3.2.1-foss-2018b |  | ✓ |  | ✓ |
HMMER/3.2.1-gompi-2019b | ✓ |  |  |  |
HMMER/3.2.1-iimpi-2019b |  |  |  | ✓ |
HMMER/3.2.1-intel-2018b |  |  |  | ✓ |
HTSeq/0.11.1-foss-2018b-Python-3.6.6 |  | ✓ |  |  |
HTSlib/1.10.2-GCC-8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
HTSlib/1.10.2-GCC-9.3.0 | ✓ |  |  |  |
HTSlib/1.11-GCC-10.2.0 | ✓ | ✓ | ✓ | ✓ | ✓
HTSlib/1.9-foss-2018b |  | ✓ |  | ✓ |
HTSlib/1.9-intel-2018b |  |  |  | ✓ |
hypothesis/4.44.2-GCCcore-8.3.0-Python-3.7.4 |  |  | ✓ | ✓ | ✓
hypothesis/5.41.2-GCCcore-10.2.0 |  |  | ✓ |  | ✓
Hypre/2.11.1-intel-2016a |  |  |  | ✓ |
Hypre/2.18.2-foss-2019b |  |  |  | ✓ |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
icc/2016.2.181-GCC-4.9.3-2.25 |  |  |  | ✓ |
icc/2016.3.210-GCC-4.9.3-2.25 |  |  |  |  | ✓
icc/2016.3.210-GCC-5.4.0-2.26 |  |  | ✓ | ✓ | ✓
icc/2017.4.196-GCC-6.4.0-2.28 |  | ✓ | ✓ | ✓ | ✓
icc/2018.1.163-GCC-6.4.0-2.28 |  |  | ✓ | ✓ |
icc/2018.2.199-GCC-6.4.0-2.28 |  |  |  |  | ✓
icc/2018.3.222-GCC-7.3.0-2.30 |  | ✓ | ✓ | ✓ | ✓
iccifortcuda/2017.4.196-GCC-6.4.0-2.28 |  |  |  |  | ✓
ICU/61.1-GCCcore-6.4.0 |  |  |  | ✓ | ✓
ICU/61.1-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
ICU/64.2-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
ICU/66.1-GCCcore-9.3.0 |  | ✓ |  | ✓ |
ICU/67.1-GCCcore-10.2.0 |  | ✓ | ✓ |  | ✓
IDBA-UD/1.1.3-GCC-8.3.0 |  | ✓ |  |  |
ifort/2016.2.181-GCC-4.9.3-2.25 |  |  |  | ✓ |
ifort/2016.3.210-GCC-4.9.3-2.25 |  |  |  |  | ✓
ifort/2016.3.210-GCC-5.4.0-2.26 |  |  | ✓ | ✓ | ✓
ifort/2017.4.196-GCC-6.4.0-2.28 |  | ✓ | ✓ | ✓ | ✓
ifort/2018.1.163-GCC-6.4.0-2.28 |  |  | ✓ | ✓ |
ifort/2018.2.199-GCC-6.4.0-2.28 |  |  |  |  | ✓
ifort/2018.3.222-GCC-7.3.0-2.30 |  | ✓ | ✓ | ✓ | ✓
igraph/0.7.1-foss-2018b |  | ✓ |  |  |
IGV/2.8.0-Java-11 |  |  |  | ✓ |
iimpi/2016.02-GCC-4.9.3-2.25 |  |  |  | ✓ |
iimpi/2016b |  |  | ✓ | ✓ | ✓
iimpi/2017b |  | ✓ | ✓ | ✓ | ✓
iimpi/2018a |  |  | ✓ | ✓ |
iimpi/2018b |  | ✓ | ✓ | ✓ | ✓
iimpi/2019b |  | ✓ | ✓ | ✓ | ✓
iimpi/2020a | ✓ | ✓ |  | ✓ |
iimpi/2020b | ✓ | ✓ | ✓ |  | ✓
ImageMagick/7.0.10-1-GCCcore-9.3.0 |  | ✓ |  | ✓ |
ImageMagick/7.0.10-35-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
ImageMagick/7.0.8-11-GCCcore-7.3.0 |  | ✓ | ✓ |  | ✓
ImageMagick/7.0.9-5-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
imkl/11.3.2.181-iimpi-2016.02-GCC-4.9.3-2.25 |  |  |  | ✓ |
imkl/11.3.3.210-iimpi-2016b |  |  | ✓ | ✓ | ✓
imkl/2017.3.196-iimpi-2017b |  | ✓ | ✓ | ✓ | ✓
imkl/2017.3.196-iompi-2017b |  | ✓ |  |  |
imkl/2018.1.163-iimpi-2018a |  |  | ✓ | ✓ |
imkl/2018.3.222-iimpi-2018b |  | ✓ | ✓ | ✓ | ✓
imkl/2018.3.222-iompi-2018b |  |  |  | ✓ |
imkl/2019.5.281-iimpi-2019b |  | ✓ | ✓ | ✓ | ✓
imkl/2020.1.217-iimpi-2020a | ✓ | ✓ |  | ✓ |
imkl/2020.4.304-iimpi-2020b | ✓ | ✓ | ✓ |  | ✓
impi/2017.3.196-iccifort-2017.4.196-GCC-6.4.0-2.28 |  | ✓ | ✓ | ✓ | ✓
impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28 |  |  | ✓ | ✓ |
impi/2018.3.222-iccifort-2018.3.222-GCC-7.3.0-2.30 |  | ✓ | ✓ | ✓ | ✓
impi/2018.5.288-iccifort-2019.5.281 |  | ✓ | ✓ | ✓ | ✓
impi/2019.7.217-iccifort-2020.1.217 | ✓ | ✓ |  | ✓ |
impi/2019.9.304-iccifort-2020.4.304 | ✓ | ✓ | ✓ |  | ✓
impi/5.1.3.181-iccifort-2016.2.181-GCC-4.9.3-2.25 |  |  |  | ✓ |
impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26 |  |  | ✓ | ✓ | ✓
intel-compilers/2021.1.2 | ✓ |  |  |  |
intel/2016a |  |  |  | ✓ |
intel/2016a_mpif90_renamed |  |  |  | ✓ |
intel/2016b |  |  | ✓ | ✓ | ✓
intel/2016b_mpif90_renamed |  |  |  | ✓ |
intel/2017b |  | ✓ | ✓ | ✓ | ✓
intel/2017b_mpif90_renamed |  |  |  | ✓ |
intel/2018a |  |  | ✓ | ✓ |
intel/2018a_mpif90_renamed |  |  |  | ✓ |
intel/2018b |  | ✓ | ✓ | ✓ | ✓
intel/2019b |  | ✓ | ✓ | ✓ | ✓
intel/2020a | ✓ | ✓ |  | ✓ |
intel/2020b | ✓ | ✓ | ✓ |  | ✓
intltool/0.51.0-foss-2016b-Perl-5.24.0 |  |  |  | ✓ |
intltool/0.51.0-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
intltool/0.51.0-GCCcore-6.4.0-Perl-5.26.0 |  | ✓ | ✓ | ✓ | ✓
intltool/0.51.0-GCCcore-6.4.0-Perl-5.26.1 |  | ✓ | ✓ | ✓ |
intltool/0.51.0-GCCcore-7.3.0-Perl-5.28.0 |  | ✓ | ✓ | ✓ | ✓
intltool/0.51.0-GCCcore-8.2.0 |  |  |  | ✓ |
intltool/0.51.0-GCCcore-8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
intltool/0.51.0-GCCcore-9.3.0 | ✓ | ✓ |  | ✓ |
iomkl/2017b |  | ✓ |  |  |
iomkl/2018b |  |  |  | ✓ |
iompi/2017b |  | ✓ |  |  |
iompi/2018b |  |  |  | ✓ |
ipyparallel/6.2.2-foss-2019b-Python-3.7.4 |  |  |  | ✓ |
IPython/5.8.0-foss-2018b-Python-2.7.15 |  | ✓ |  |  |
IPython/7.15.0-foss-2020a-Python-3.8.2 | ✓ |  |  |  |
IPython/7.18.1-GCCcore-10.2.0 | ✓ |  |  |  |
IPython/7.2.0-foss-2018b-Python-3.6.6 |  |  | ✓ |  | ✓
IPython/7.2.0-fosscuda-2018b-Python-3.6.6 |  |  | ✓ |  |
IPython/7.2.0-intel-2018b-Python-3.6.6 |  |  | ✓ |  | ✓
IPython/7.9.0-foss-2019b-Python-3.7.4 | ✓ | ✓ | ✓ | ✓ | ✓
IPython/7.9.0-fosscuda-2019b-Python-3.7.4 |  | ✓ | ✓ |  | ✓
ITSTool/2.0.5-foss-2018b-Python-2.7.15 |  |  |  |  | ✓

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
JAGS/4.3.0-foss-2020a |  | ✓ |  | ✓ |
JasPer/1.900.1-foss-2016b |  |  |  | ✓ |
JasPer/1.900.1-GCCcore-6.4.0 |  |  | ✓ | ✓ |
JasPer/1.900.1-intel-2016a |  |  |  | ✓ |
JasPer/1.900.1-intel-2018a |  |  |  | ✓ |
JasPer/2.0.14-GCCcore-10.2.0 | ✓ | ✓ |  |  |
JasPer/2.0.14-GCCcore-6.4.0 |  |  |  | ✓ | ✓
JasPer/2.0.14-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
JasPer/2.0.14-GCCcore-8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
JasPer/2.0.14-GCCcore-9.3.0 | ✓ | ✓ |  | ✓ |
JasPer/2.0.24-GCCcore-10.2.0 | ✓ |  | ✓ |  | ✓
Java/1.8.0_121 |  |  |  | ✓ |
Java/1.8.0_141 |  |  |  | ✓ |
Java/1.8.0_152 |  |  | ✓ | ✓ | ✓
Java/1.8.0_162 |  |  | ✓ | ✓ | ✓
Java/1.8.0_192 |  |  | ✓ | ✓ |
Java/1.8.0_241 |  |  | ✓ | ✓ | ✓
Java/1.8.0_31 |  | ✓ |  | ✓ |
Java/1.8.0_92 |  |  |  | ✓ |
Java/11.0.2 |  | ✓ | ✓ | ✓ | ✓
Jellyfish/1.1.12-foss-2018b |  |  |  | ✓ |
Jellyfish/2.2.10-foss-2018b |  | ✓ |  | ✓ |
jemalloc/5.1.0-GCCcore-7.3.0 |  | ✓ |  | ✓ |
jemalloc/5.2.1-GCCcore-9.3.0 |  |  |  | ✓ |
JsonCpp/1.9.3-GCCcore-8.3.0 |  |  | ✓ |  | ✓
JsonCpp/1.9.4-GCCcore-10.2.0 | ✓ |  |  |  |
Julia/1.5.1-linux-x86_64 |  |  | ✓ |  |
JUnit/4.12-Java-1.8 |  |  | ✓ |  |
JUnit/4.12-Java-1.8.0_92 |  |  |  | ✓ |
JupyterHub/1.1.0-GCCcore-10.2.0 | ✓ |  |  |  |
JupyterLab/1.2.5-fosscuda-2019b-Python-3.7.4 |  |  | ✓ |  | ✓
JupyterLab/2.2.8-GCCcore-10.2.0 | ✓ |  |  |  |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
Keras/2.2.4-foss-2018b-Python-3.6.6 |  |  |  | ✓ |
Keras/2.2.4-fosscuda-2018b-Python-3.6.6 |  |  | ✓ |  |
kim-api/2.1.3-foss-2019b |  | ✓ |  |  |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
LAME/3.100-GCCcore-10.2.0 | ✓ |  |  |  |
LAME/3.100-GCCcore-7.3.0 |  | ✓ | ✓ |  | ✓
LAME/3.100-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
LAME/3.100-GCCcore-9.3.0 | ✓ |  |  | ✓ |
LAMMPS/3Mar2020-foss-2019b-Python-3.7.4-kokkos |  | ✓ |  |  |
LEMON/1.3.1-GCC-8.2.0-2.31.1 |  |  |  | ✓ |
libarchive/3.4.3-GCCcore-10.2.0 | ✓ | ✓ | ✓ | ✓ | ✓
libcerf/1.13-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
libcerf/1.13-GCCcore-9.3.0 | ✓ |  |  |  |
libcerf/1.14-GCCcore-10.2.0 | ✓ |  |  |  |
libcerf/1.5-GCCcore-6.4.0 |  |  | ✓ | ✓ |
libcerf/1.7-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
libdap/3.19.1-intel-2017b |  |  | ✓ | ✓ |
libdap/3.20.3-GCCcore-7.3.0 |  |  | ✓ | ✓ |
libdap/3.20.6-GCCcore-8.3.0 |  |  |  | ✓ |
libepoxy/1.5.4-GCCcore-8.3.0 |  |  | ✓ |  |
libfabric/1.11.0-GCCcore-10.2.0 |  | ✓ | ✓ | ✓ | ✓
libfabric/1.11.0-GCCcore-9.3.0 |  | ✓ |  |  |
libFLAME/2.2-GCCcore-9.3.0-amd | ✓ |  |  |  |
libFLAME/5.2.0-GCC-9.3.0 | ✓ |  |  |  |
libFLAME/5.2.0-GCCcore-9.3.0 | ✓ |  |  |  |
libgd/2.2.5-GCCcore-6.4.0 |  |  | ✓ | ✓ |
libgd/2.2.5-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
libgd/2.2.5-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
libgd/2.3.0-GCCcore-10.2.0 | ✓ |  |  |  |
libgd/2.3.0-GCCcore-9.3.0 | ✓ |  |  |  |
libgeotiff/1.4.2-foss-2018b |  | ✓ | ✓ | ✓ | ✓
libgeotiff/1.4.2-intel-2018b |  |  |  |  | ✓
libgeotiff/1.5.1-GCCcore-8.3.0 | ✓ |  | ✓ |  |
libgeotiff/1.5.1-GCCcore-9.3.0 | ✓ | ✓ |  | ✓ |
libgeotiff/1.6.0-GCCcore-10.2.0 |  |  | ✓ |  | ✓
libglade/2.6.4-foss-2018b |  |  |  |  | ✓
libGLU/9.0.0-foss-2016b |  |  |  | ✓ |
libGLU/9.0.0-foss-2017b |  |  | ✓ | ✓ | ✓
libGLU/9.0.0-foss-2018a |  |  | ✓ | ✓ |
libGLU/9.0.0-foss-2018b |  |  | ✓ | ✓ | ✓
libGLU/9.0.0-fosscuda-2018b |  |  | ✓ |  |
libGLU/9.0.0-intel-2017b |  |  |  | ✓ | ✓
libGLU/9.0.0-intel-2018b |  |  | ✓ |  | ✓
libGLU/9.0.1-GCCcore-10.2.0 |  | ✓ | ✓ |  | ✓
libGLU/9.0.1-GCCcore-8.3.0 |  |  | ✓ | ✓ | ✓
libGLU/9.0.1-GCCcore-9.3.0 |  |  |  | ✓ |
libglvnd/1.2.0-GCCcore-9.3.0 |  | ✓ |  | ✓ |
libglvnd/1.3.2-GCCcore-10.2.0 |  | ✓ | ✓ |  | ✓
libgpuarray/0.7.6-fosscuda-2018b-Python-3.6.6 |  |  | ✓ |  |
libgpuarray/0.7.6-fosscuda-2019b-Python-3.7.4 |  |  |  |  | ✓
libharu/2.3.0-GCCcore-7.3.0 |  | ✓ |  |  |
libharu/2.3.0-intel-2018b |  |  |  | ✓ |
Libint/1.1.6-foss-2018b |  |  | ✓ |  |
Libint/1.1.6-intel-2017b |  | ✓ |  | ✓ |
Libint/1.1.6-intel-2018a |  |  | ✓ |  |
Libint/1.1.6-intel-2018b |  |  |  |  | ✓
Libint/2.0.3-foss-2018b |  |  |  |  | ✓
Libint/2.6.0-iimpi-2020a-lmax-6-cp2k |  |  |  | ✓ |
libmatheval/1.1.11-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ |
libmatheval/1.1.11-GCCcore-7.3.0 |  |  |  |  | ✓
libmatheval/1.1.11-GCCcore-8.3.0 |  | ✓ |  |  |
libmatheval/1.1.11-GCCcore-9.3.0 |  |  |  | ✓ |
libogg/1.3.4-GCCcore-10.2.0 |  |  | ✓ |  | ✓
libsigc++/2.10.2-GCCcore-8.3.0 |  | ✓ |  | ✓ |
libsndfile/1.0.28-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
libsndfile/1.0.28-GCCcore-6.4.0 |  |  |  | ✓ | ✓
libsndfile/1.0.28-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
libsndfile/1.0.28-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
libsndfile/1.0.28-GCCcore-9.3.0 |  | ✓ |  | ✓ |
libsodium/1.0.16-GCCcore-7.3.0 |  | ✓ | ✓ |  | ✓
libsodium/1.0.18-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
libspatialindex/1.8.5-foss-2016b |  |  |  | ✓ |
libspatialite/4.3.0a-foss-2016b |  |  |  | ✓ |
libtirpc/1.1.4-GCCcore-7.3.0 |  |  |  | ✓ |
libtirpc/1.1.4-GCCcore-8.3.0 |  |  |  | ✓ |
libtirpc/1.2.6-GCCcore-9.3.0 |  | ✓ |  |  |
libtirpc/1.3.1-GCCcore-10.2.0 |  |  | ✓ |  | ✓
libunistring/0.9.10-GCCcore-7.3.0 |  |  |  |  | ✓
libunistring/0.9.10-GCCcore-8.3.0 |  | ✓ |  |  |
libunistring/0.9.10-GCCcore-9.3.0 |  |  |  | ✓ |
libunistring/0.9.7-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ |
libvorbis/1.3.7-GCCcore-10.2.0 |  |  | ✓ |  | ✓
libxc/2.2.3-intel-2017b |  |  | ✓ |  |
libxc/3.0.0-GCC-5.4.0-2.26 |  |  |  | ✓ |
libxc/3.0.0-intel-2016a |  |  |  | ✓ |
libxc/3.0.0-intel-2017b |  | ✓ |  | ✓ |
libxc/3.0.1-intel-2018a |  |  | ✓ | ✓ |
libxc/3.0.1-intel-2018b |  | ✓ | ✓ | ✓ |
libxc/4.0.1-intel-2017b |  |  | ✓ |  |
libxc/4.2.3-foss-2018b |  |  | ✓ | ✓ | ✓
libxc/4.2.3-intel-2018a |  |  | ✓ |  |
libxc/4.2.3-intel-2018b |  |  | ✓ | ✓ | ✓
libxc/4.3.4-GCC-10.2.0 | ✓ |  |  |  |
libxc/4.3.4-GCC-8.3.0 | ✓ |  |  |  |
libxc/4.3.4-GCC-9.3.0 | ✓ |  |  |  |
libxc/4.3.4-iccifort-2019.5.281 |  |  | ✓ | ✓ | ✓
libxc/4.3.4-iccifort-2020.1.217 | ✓ | ✓ |  | ✓ |
libxc/4.3.4-iccifort-2020.4.304 | ✓ | ✓ |  |  |
libxml++/2.40.1-GCCcore-8.3.0 |  | ✓ |  | ✓ |
libxml2-python/2.9.8-foss-2018b-Python-2.7.15 |  |  |  |  | ✓
libxslt/1.1.32-GCCcore-6.4.0 |  | ✓ |  |  |
libxslt/1.1.34-GCCcore-8.3.0 |  |  |  | ✓ |
libxsmm/1.16.1-iccifort-2020.1.217 |  |  |  | ✓ |
libxsmm/1.8.2-intel-2017b |  | ✓ |  | ✓ |
libxsmm/1.8.3-foss-2018b |  |  | ✓ |  | ✓
libxsmm/1.8.3-intel-2018a |  |  | ✓ |  |
libxsmm/1.8.3-intel-2018b |  |  |  |  | ✓
libyaml/0.1.7 |  |  |  | ✓ |
libyaml/0.1.7-GCCcore-6.4.0 |  | ✓ |  |  |
libyaml/0.2.1-GCCcore-7.3.0 |  |  | ✓ | ✓ |
libyaml/0.2.2-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
LLVM/10.0.1-GCCcore-10.2.0 | ✓ |  |  |  |
LLVM/11.0.0-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
LLVM/3.8.0-intel-2016a |  |  |  | ✓ |
LLVM/3.8.1-foss-2016b |  |  |  | ✓ |
LLVM/5.0.0-foss-2017b |  |  | ✓ | ✓ | ✓
LLVM/5.0.0-intel-2017b |  |  |  | ✓ | ✓
LLVM/5.0.1-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ |
LLVM/6.0.0-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
LLVM/8.0.1-GCCcore-8.3.0 | ✓ | ✓ |  | ✓ |
LLVM/9.0.0-GCCcore-8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
LLVM/9.0.1-GCCcore-9.3.0 | ✓ | ✓ |  | ✓ |
LMfit/1.0.1-foss-2020a-Python-3.8.2 | ✓ |  |  | ✓ |
lpsolve/5.5.2.5-foss-2018b |  | ✓ |  |  |
lpsolve/5.5.2.5-intel-2018b |  |  |  | ✓ |
Lua/5.1.5-GCCcore-7.3.0 |  |  |  |  | ✓
Lua/5.1.5-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
Lua/5.3.5-GCCcore-9.3.0 | ✓ |  |  |  |
Lua/5.4.2-GCCcore-10.2.0 | ✓ |  |  |  |
lxml/4.1.1-intel-2017b-Python-2.7.14 |  | ✓ |  |  |
lz4/1.9.2-GCCcore-10.2.0 |  | ✓ | ✓ |  | ✓
lz4/1.9.2-GCCcore-9.3.0 |  | ✓ |  | ✓ |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
MACS2/2.2.5-foss-2018b-Python-3.6.6 |  | ✓ |  |  |
MAFFT/7.427-intel-2018b-with-extensions |  |  |  | ✓ |
magma/2.5.1-fosscuda-2019b |  |  | ✓ |  |
magma/2.5.4-fosscuda-2019b |  |  | ✓ |  | ✓
make/4.3-GCC-9.3.0 | ✓ |  |  |  |
make/4.3-GCCcore-9.3.0 | ✓ |  |  |  |
makedepend/1.0.5-GCCcore-6.4.0 |  |  | ✓ |  |
makedepend/1.0.5-intel-2016a |  |  |  | ✓ |
makeinfo/6.7-GCCcore-9.3.0 |  | ✓ |  |  |
Mako/1.0.4-foss-2016b-Python-2.7.12 |  |  |  | ✓ |
Mako/1.0.7-foss-2017b-Python-2.7.14 |  |  | ✓ | ✓ | ✓
Mako/1.0.7-foss-2018a-Python-2.7.14 |  | ✓ | ✓ | ✓ |
Mako/1.0.7-foss-2018b-Python-2.7.15 |  | ✓ | ✓ | ✓ | ✓
Mako/1.0.7-fosscuda-2018b-Python-2.7.15 |  |  | ✓ |  |
Mako/1.0.7-fosscuda-2018b-Python-3.6.6 |  |  | ✓ |  |
Mako/1.0.7-intel-2017b-Python-2.7.14 |  |  |  | ✓ | ✓
Mako/1.0.7-intel-2018b-Python-2.7.15 |  | ✓ | ✓ |  | ✓
Mako/1.1.0-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
Mako/1.1.2-GCCcore-9.3.0 |  | ✓ |  | ✓ |
Mako/1.1.3-GCCcore-10.2.0 |  | ✓ | ✓ |  | ✓
MariaDB-connector-c/3.1.7-GCCcore-9.3.0 | ✓ | ✓ |  | ✓ |
MaSuRCA/3.3.1-foss-2018b-Perl-5.28.0 |  | ✓ |  |  |
MathGL/2.4.1-foss-2018a |  | ✓ |  |  |
matplotlib/2.1.1-intel-2017b-Python-2.7.14 |  | ✓ |  |  |
matplotlib/2.2.3-foss-2018b-Python-2.7.15 |  | ✓ |  | ✓ |
matplotlib/3.0.0-foss-2018b-Python-3.6.6 |  | ✓ | ✓ | ✓ | ✓
matplotlib/3.0.0-fosscuda-2018b-Python-3.6.6 |  |  | ✓ |  |
matplotlib/3.0.0-intel-2018b-Python-3.6.6 |  |  | ✓ |  | ✓
matplotlib/3.0.2-foss-2018b-Python-3.6.6 |  |  |  |  | ✓
matplotlib/3.1.1-foss-2019b-Python-3.7.4 | ✓ | ✓ | ✓ | ✓ | ✓
matplotlib/3.1.1-fosscuda-2019b-Python-3.7.4 |  | ✓ | ✓ |  | ✓
matplotlib/3.2.1-foss-2020a-Python-3.8.2 | ✓ | ✓ |  | ✓ |
matplotlib/3.3.3-foss-2020b | ✓ |  |  |  |
MaxQuant/1.6.17.0-foss-2018b |  | ✓ |  |  |
MCL/14.137-GCCcore-8.3.0 |  | ✓ |  | ✓ |
MCR/R2014b |  | ✓ |  |  |
MCR/R2015a |  | ✓ |  |  |
MCR/R2015b |  | ✓ |  |  |
MCR/R2016a |  | ✓ |  |  |
MCR/R2016b |  | ✓ |  |  |
MCR/R2017a |  | ✓ |  |  |
MCR/R2017b |  | ✓ |  |  |
MCR/R2018a |  | ✓ |  |  |
MCR/R2018b |  | ✓ |  |  | ✓
Meson/0.50.0-GCCcore-8.2.0-Python-3.7.2 |  |  |  | ✓ |
Meson/0.51.2-GCCcore-8.3.0-Python-3.7.4 |  | ✓ | ✓ | ✓ | ✓
Meson/0.53.2-GCCcore-9.3.0-Python-3.8.2 |  |  |  | ✓ |
Meson/0.55.1-GCCcore-9.3.0-Python-3.8.2 |  | ✓ |  |  |
Meson/0.55.3-GCCcore-10.2.0 |  | ✓ | ✓ |  | ✓
MetaGeneAnnotator/20080819-x86-64 |  |  |  | ✓ |
METIS/5.1.0-foss-2016b |  |  |  | ✓ |
METIS/5.1.0-foss-2018b |  | ✓ | ✓ | ✓ | ✓
METIS/5.1.0-GCCcore-10.2.0 | ✓ |  |  |  |
METIS/5.1.0-GCCcore-6.4.0 |  |  | ✓ | ✓ |
METIS/5.1.0-GCCcore-7.3.0 |  |  |  | ✓ |
METIS/5.1.0-GCCcore-8.3.0 |  |  | ✓ | ✓ | ✓
METIS/5.1.0-GCCcore-9.3.0 | ✓ |  |  | ✓ |
METIS/5.1.0-intel-2016a |  |  |  | ✓ |
Minia/3.2.4-foss-2018b |  | ✓ |  |  |
Miniconda3/4.7.10 | ✓ |  |  | ✓ |
MLxtend/0.17.3-foss-2020a-Python-3.8.2 | ✓ |  |  |  |
MMseqs2/10-6d92c-gompi-2019b |  | ✓ |  |  |
MMseqs2/10-6d92c-iimpi-2019b |  |  |  | ✓ |
MMseqs2/11-e1a1c-iimpi-2019b |  | ✓ |  |  |
molmod/1.4.5-foss-2019b-Python-3.7.4 |  | ✓ |  |  |
MOLPRINT2D/1.2 |  | ✓ |  |  |
Mono/6.4.0.198-foss-2018b |  | ✓ |  |  |
mosdepth/0.2.4-foss-2018b |  |  |  | ✓ |
Mothur/1.41.0-foss-2018b-Python-2.7.15 |  |  |  | ✓ | ✓
MPFR/3.1.4-foss-2016b |  |  |  | ✓ |
MPFR/3.1.5-GCCcore-6.4.0 |  |  | ✓ | ✓ |
MPFR/4.0.1-GCCcore-6.4.0 |  |  | ✓ |  |
MPFR/4.0.1-GCCcore-7.3.0 |  |  | ✓ | ✓ | ✓
MPFR/4.0.2-GCCcore-8.3.0 |  |  | ✓ | ✓ | ✓
MPFR/4.0.2-GCCcore-9.3.0 | ✓ |  |  | ✓ |
MPFR/4.1.0-GCCcore-10.2.0 | ✓ |  |  |  |
MPICH/3.3.2-GCC-10.2.0 | ✓ |  |  |  |
mpmath/0.19-intel-2016a-Python-2.7.11 |  |  |  | ✓ |
MUMPS/5.1.2-foss-2017b-metis |  |  |  | ✓ |
MUMPS/5.2.1-foss-2019b-metis |  |  |  | ✓ |
MUSCLE/3.8.1551-GCC-8.3.0 | ✓ |  |  |  |
MUSCLE/3.8.31-foss-2019b |  |  |  | ✓ |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
NAMD/2.12-foss-2016b-CUDA-8.0.61 |  |  |  |  | ✓
NAMD/2.12-foss-2017b-mpi |  | ✓ |  |  |
NAMD/2.12-intel-2017b-mpi |  | ✓ | ✓ | ✓ |
NAMD/2.13-foss-2018b-mpi |  | ✓ | ✓ | ✓ |
NAMD/2.13-fosscuda-2018b |  | ✓ |  |  | ✓
NAMD/2.14-foss-2019b-mpi | ✓ |  |  |  |
NAMD/2.9-intel-2017b-mpi |  |  |  | ✓ |
NASM/2.11.08-intel-2016a |  |  |  | ✓ |
NASM/2.12.01-intel-2016a |  |  |  | ✓ |
NASM/2.12.02-foss-2016b |  |  |  | ✓ |
NASM/2.12.02-foss-2018a |  |  |  | ✓ |
NASM/2.12.02-intel-2016b |  |  |  | ✓ |
NASM/2.12.02-intel-2018a |  |  |  | ✓ |
NASM/2.13.01-GCCcore-6.4.0 |  |  | ✓ | ✓ | ✓
NASM/2.13.03-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ |
NASM/2.13.03-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
NASM/2.14.02-GCCcore-8.2.0 |  |  |  | ✓ |
NASM/2.14.02-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
NASM/2.14.02-GCCcore-9.3.0 |  | ✓ |  | ✓ |
NASM/2.15.05-GCCcore-10.2.0 |  | ✓ | ✓ |  | ✓
ncbi-vdb/2.9.3-foss-2018b |  |  |  | ✓ |
NCCL/2.3.7-fosscuda-2018b |  |  | ✓ |  | ✓
NCCL/2.4.8-gcccuda-2019b |  |  | ✓ |  | ✓
ncdf4/1.17-foss-2020a-R-4.0.0 | ✓ | ✓ |  | ✓ |
ncdf4/1.17-foss-2020b-R-4.0.3 | ✓ | ✓ |  |  |
NCL/6.4.0-intel-2017b |  |  | ✓ |  |
NCO/4.7.1-intel-2017b |  |  | ✓ | ✓ |
NCO/4.7.9-intel-2018b |  |  | ✓ | ✓ |
NCO/4.9.1-intel-2019b |  |  |  | ✓ |
NCO/4.9.3-gompi-2019b | ✓ |  |  |  |
ncview/2.1.7-foss-2019b | ✓ |  |  |  |
ncview/2.1.7-intel-2017b |  |  |  | ✓ |
ncview/2.1.7-intel-2019b |  |  |  | ✓ |
netCDF-C++/4.2-foss-2018a |  |  |  | ✓ |
netCDF-C++4/4.3.0-intel-2017b-HDF5-1.8.19 |  |  | ✓ | ✓ |
netCDF-C++4/4.3.0-intel-2018b |  |  | ✓ | ✓ |
netCDF-C++4/4.3.1-iimpi-2019b |  |  |  | ✓ |
netCDF-Fortran/4.4.4-foss-2016b |  |  |  | ✓ |
netCDF-Fortran/4.4.4-foss-2017b |  |  |  |  | ✓
netCDF-Fortran/4.4.4-foss-2018b |  |  | ✓ | ✓ | ✓
netCDF-Fortran/4.4.4-intel-2016a |  |  |  | ✓ |
netCDF-Fortran/4.4.4-intel-2017b |  |  | ✓ | ✓ |
netCDF-Fortran/4.4.4-intel-2017b-HDF5-1.8.19 |  |  | ✓ | ✓ |
netCDF-Fortran/4.4.4-intel-2018a |  |  |  | ✓ |
netCDF-Fortran/4.4.4-intel-2018b |  | ✓ | ✓ | ✓ |
netCDF-Fortran/4.5.2-gompi-2019b | ✓ | ✓ |  | ✓ |
netCDF-Fortran/4.5.2-gompi-2020a | ✓ |  |  |  |
netCDF-Fortran/4.5.2-iimpi-2019b |  |  | ✓ | ✓ | ✓
netCDF-Fortran/4.5.2-iimpi-2020a |  | ✓ |  | ✓ |
netCDF-Fortran/4.5.3-gompi-2020b | ✓ |  |  |  |
netCDF-Fortran/4.5.3-iimpi-2020b | ✓ | ✓ |  |  |
netCDF/4.4.0-foss-2018a |  |  |  | ✓ |
netCDF/4.4.0-intel-2016a |  |  |  | ✓ |
netCDF/4.4.1-foss-2016b |  |  |  | ✓ |
netCDF/4.4.1.1-foss-2016b |  |  |  | ✓ |
netCDF/4.4.1.1-foss-2017b |  |  |  |  | ✓
netCDF/4.4.1.1-intel-2016a |  |  |  | ✓ |
netCDF/4.4.1.1-intel-2017b |  |  | ✓ | ✓ |
netCDF/4.4.1.1-intel-2017b-HDF5-1.8.19 |  |  | ✓ | ✓ |
netCDF/4.4.1.1-intel-2018a |  |  |  | ✓ |
netCDF/4.5.0-foss-2017b |  |  |  | ✓ | ✓
netCDF/4.5.0-foss-2018a |  |  |  | ✓ |
netCDF/4.6.0-intel-2018a |  |  | ✓ | ✓ |
netCDF/4.6.1-foss-2018b |  | ✓ | ✓ | ✓ | ✓
netCDF/4.6.1-intel-2018b |  | ✓ | ✓ | ✓ | ✓
netCDF/4.7.1-gompi-2019b | ✓ | ✓ | ✓ | ✓ | ✓
netCDF/4.7.1-gompic-2019b |  |  | ✓ |  |
netCDF/4.7.1-iimpi-2019b |  |  | ✓ | ✓ | ✓
netCDF/4.7.4-gompi-2020a | ✓ | ✓ |  | ✓ |
netCDF/4.7.4-gompi-2020b | ✓ | ✓ | ✓ |  | ✓
netCDF/4.7.4-iimpi-2020a |  | ✓ |  | ✓ |
netCDF/4.7.4-iimpi-2020b | ✓ | ✓ |  |  |
netcdf4-python/1.4.0-intel-2018a-Python-3.6.4 |  |  |  | ✓ |
NetLogo/6.1.1-64 |  | ✓ |  |  |
nettle/3.2-foss-2016b |  |  |  | ✓ |
nettle/3.3-GCCcore-6.4.0 |  |  | ✓ | ✓ | ✓
nettle/3.4-foss-2018a |  | ✓ | ✓ | ✓ |
nettle/3.4-foss-2018b |  | ✓ | ✓ | ✓ | ✓
nettle/3.4-fosscuda-2018b |  |  | ✓ |  |
nettle/3.4-intel-2018b |  | ✓ | ✓ |  | ✓
nettle/3.5.1-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
nettle/3.6-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
nettle/3.6-GCCcore-9.3.0 |  | ✓ |  |  |
networkx/2.4-foss-2019b-Python-3.7.4 |  | ✓ |  |  |
networkx/2.4-foss-2020a-Python-3.8.2 | ✓ |  |  |  |
networkx/2.4-fosscuda-2020a-Python-3.8.2 |  | ✓ |  |  |
Nextflow/20.04.1 |  |  |  | ✓ |
NGS/2.9.3-foss-2018b-Java-1.8 |  |  |  | ✓ |
Nim/0.19.2-GCCcore-7.3.0 |  |  |  | ✓ |
Ninja/1.10.0-GCCcore-9.3.0 |  | ✓ |  | ✓ |
Ninja/1.10.1-GCCcore-10.2.0 |  | ✓ | ✓ |  | ✓
Ninja/1.9.0-fosscuda-2019b |  |  | ✓ |  |
Ninja/1.9.0-GCCcore-8.2.0 |  |  |  | ✓ |
Ninja/1.9.0-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
NLopt/2.4.2-foss-2017b |  |  |  | ✓ | ✓
NLopt/2.4.2-foss-2018a |  | ✓ |  |  |
NLopt/2.4.2-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
NLopt/2.6.1-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
NLopt/2.6.1-GCCcore-9.3.0 |  | ✓ |  | ✓ |
NLopt/2.6.2-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
nodejs/12.19.0-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
NSPR/4.20-GCCcore-6.4.0 |  |  | ✓ |  |
NSPR/4.20-GCCcore-7.3.0 |  | ✓ | ✓ |  | ✓
NSPR/4.21-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
NSPR/4.25-GCCcore-9.3.0 | ✓ |  |  | ✓ |
NSPR/4.29-GCCcore-10.2.0 | ✓ |  |  |  |
NSS/3.39-GCCcore-6.4.0 |  |  | ✓ |  |
NSS/3.39-GCCcore-7.3.0 |  | ✓ | ✓ |  | ✓
NSS/3.45-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
NSS/3.51-GCCcore-9.3.0 | ✓ |  |  | ✓ |
NSS/3.57-GCCcore-10.2.0 | ✓ |  |  |  |
nsync/1.24.0-GCCcore-10.2.0 | ✓ |  |  |  |
nsync/1.24.0-GCCcore-8.3.0 |  |  | ✓ |  | ✓
numba/0.47.0-foss-2019b-Python-3.7.4 | ✓ | ✓ |  | ✓ |
numba/0.50.0-foss-2020a-Python-3.8.2 |  | ✓ |  | ✓ |
numba/0.52.0-foss-2020b | ✓ |  |  |  |
numexpr/2.7.1-foss-2019b-Python-3.7.4 |  | ✓ |  |  |
numexpr/2.7.1-foss-2020a-Python-3.8.2 | ✓ | ✓ |  | ✓ |
numpy/1.8.2-intel-2016a-Python-2.7.11 |  |  |  | ✓ |
NWChem/6.8.revision47-intel-2017b-2017-12-14-Python-2.7.14 |  |  | ✓ |  |
NWChem/6.8.revision47-intel-2018a-2017-12-14-Python-2.7.14 |  |  | ✓ |  |
NWChem/7.0.0-intel-2019b-Python-3.7.4 |  |  | ✓ |  | ✓

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
OBITools/1.2.13-foss-2018b-Python-2.7.15 |  | ✓ |  |  |
Octave/4.2.2-foss-2018a |  |  | ✓ | ✓ |
Octave/4.4.1-foss-2018b |  |  | ✓ | ✓ |
Octave/5.1.0-foss-2019b |  |  | ✓ |  | ✓
OPARI2/2.0.5-GCCcore-9.3.0 | ✓ |  |  |  |
OpenBabel/3.1.1-gompi-2019b-Python-3.7.4 |  | ✓ |  |  |
OpenBLAS/0.2.15-GCC-4.9.3-2.25-LAPACK-3.6.0 |  |  |  | ✓ |
OpenBLAS/0.2.18-GCC-5.4.0-2.26-LAPACK-3.6.1 |  |  |  | ✓ | ✓
OpenBLAS/0.2.20-GCC-6.4.0-2.28 |  | ✓ | ✓ | ✓ | ✓
OpenBLAS/0.3.1-GCC-7.3.0-2.30 |  | ✓ | ✓ | ✓ | ✓
OpenBLAS/0.3.12-GCC-10.2.0 | ✓ | ✓ | ✓ | ✓ | ✓
OpenBLAS/0.3.5-GCC-8.2.0-2.31.1 |  |  |  | ✓ |
OpenBLAS/0.3.7-GCC-8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
OpenBLAS/0.3.9-GCC-9.3.0 | ✓ | ✓ |  | ✓ |
OpenCV/4.2.0-foss-2019b-Python-3.7.4 |  |  | ✓ |  |
OpenCV/4.2.0-fosscuda-2019b-Python-3.7.4 |  |  | ✓ |  |
OpenFOAM/4.1-foss-2016b |  |  |  | ✓ |
OpenFOAM/8-foss-2020a |  |  |  | ✓ |
OpenFOAM/8-foss-2020b | ✓ |  |  |  |
OpenFOAM/v1806-foss-2018b |  |  |  | ✓ |
OpenFOAM/v1812-foss-2018b |  |  | ✓ | ✓ | ✓
OpenFOAM/v1912-foss-2019b |  |  | ✓ |  | ✓
OpenFOAM/v1912-intel-2019b |  |  |  | ✓ |
OpenFOAM/v2012-foss-2020a | ✓ |  |  |  |
openjpeg/1.5.2-intel-2016a |  |  |  | ✓ |
openjpeg/1.5.2-intel-2018a |  |  |  | ✓ |
OpenJPEG/2.3.1-GCCcore-8.2.0 |  |  |  | ✓ |
OpenMPI/1.10.3-GCC-5.4.0-2.26 |  |  |  | ✓ | ✓
OpenMPI/2.0.2-GCC-6.3.0-2.27 |  |  | ✓ | ✓ |
OpenMPI/2.1.1-GCC-6.4.0-2.28 |  | ✓ | ✓ | ✓ | ✓
OpenMPI/2.1.1-gcccuda-2017b |  |  |  |  | ✓
OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28 |  | ✓ |  |  |
OpenMPI/2.1.2-GCC-6.4.0-2.28 |  | ✓ | ✓ | ✓ |
OpenMPI/2.1.2-gcccuda-2018a |  |  | ✓ |  |
OpenMPI/2.1.3-iccifort-2018.2.199-GCC-6.4.0-2.28 |  |  |  |  | ✓
OpenMPI/3.1.1-GCC-7.3.0-2.30 |  | ✓ | ✓ | ✓ | ✓
OpenMPI/3.1.1-gcccuda-2018b |  | ✓ | ✓ |  | ✓
OpenMPI/3.1.1-iccifort-2018.3.222-GCC-7.3.0-2.30 |  |  |  | ✓ |
OpenMPI/3.1.3-GCC-8.2.0-2.31.1 |  |  | ✓ | ✓ |
OpenMPI/3.1.4-GCC-8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
OpenMPI/3.1.4-gcccuda-2019b |  | ✓ | ✓ |  | ✓
OpenMPI/4.0.3-GCC-9.3.0 | ✓ | ✓ |  | ✓ |
OpenMPI/4.0.3-gcccuda-2020a |  | ✓ |  |  |
OpenMPI/4.0.5-GCC-10.2.0 | ✓ | ✓ | ✓ | ✓ | ✓
OpenMPI/4.0.5-GCC-9.3.0 | ✓ |  |  | ✓ |
OpenMPI/4.1.0-GCC-10.2.0 | ✓ |  |  |  |
OpenPGM/5.2.122-GCCcore-7.3.0 |  | ✓ | ✓ |  | ✓
OpenPGM/5.2.122-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
openslide-python/1.1.1-GCCcore-8.2.0 |  |  |  | ✓ |
OpenSlide/3.4.1-GCCcore-8.2.0 |  |  |  | ✓ |
ORCA/4.0.1-OpenMPI-2.0.2 |  |  | ✓ | ✓ |
ORCA/4.1.0-OpenMPI-3.1.3 |  |  | ✓ | ✓ |
ORCA/4.2.1-gompi-2019b | ✓ |  | ✓ |  | ✓
OSU-Micro-Benchmarks/5.3.2-foss-2016b |  |  |  | ✓ |
OSU-Micro-Benchmarks/5.3.2-foss-2017b |  |  | ✓ | ✓ |
OSU-Micro-Benchmarks/5.3.2-foss-2018a |  |  |  | ✓ |
OSU-Micro-Benchmarks/5.3.2-foss-2018b |  |  |  | ✓ |
OSU-Micro-Benchmarks/5.3.2-intel-2016a |  |  |  | ✓ |
OSU-Micro-Benchmarks/5.3.2-intel-2016b |  |  |  | ✓ |
OSU-Micro-Benchmarks/5.3.2-intel-2017b |  |  | ✓ | ✓ |
OSU-Micro-Benchmarks/5.3.2-intel-2018a |  |  |  | ✓ |
OSU-Micro-Benchmarks/5.6.3-gompi-2020a | ✓ |  |  |  |
OSU-Micro-Benchmarks/5.6.3-gompi-2020b | ✓ |  |  |  |
OTF2/2.2-GCCcore-9.3.0 | ✓ |  |  |  |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
p4vasp/0.3.30-foss-2018b-Python-2.7.15 |  |  |  |  | ✓
packmol/18.169-foss-2018b |  | ✓ |  |  |
packmol/18.169-intel-2018b |  | ✓ |  |  |
pandas/1.0.0-foss-2019b-Python-3.7.4 |  |  |  | ✓ |
pandas/1.1.2-foss-2020a-Python-3.8.2 |  | ✓ |  |  |
PAPI/6.0.0-GCCcore-9.3.0 | ✓ |  |  |  |
parallel/20171122-intel-2017b |  |  |  | ✓ |
parallel/20200522-GCCcore-9.3.0 |  |  |  | ✓ |
ParaView/5.2.0-foss-2016b-mpi |  |  |  | ✓ |
ParaView/5.4.1-foss-2018b-mpi |  |  | ✓ | ✓ | ✓
ParaView/5.4.1-intel-2017b-mpi |  |  |  | ✓ |
ParaView/5.4.1-intel-2017b-mpi-HDF5-1.8.19 |  |  |  | ✓ |
ParaView/5.6.2-foss-2019b-Python-3.7.4-mpi |  |  | ✓ |  | ✓
ParaView/5.6.2-intel-2019b-Python-3.7.4-mpi |  |  |  | ✓ |
ParaView/5.8.0-foss-2020a-Python-3.8.2-mpi | ✓ |  |  | ✓ |
ParaView/5.8.1-foss-2020b-mpi | ✓ |  |  |  |
ParaViewGMSH/5.6.2-foss-2019b-Python-3.7.4-mpi |  |  |  | ✓ |
ParaViewGMSH/5.7.0-foss-2019b-Python-3.7.4-mpi |  |  |  | ✓ |
ParMETIS/4.0.3-intel-2016a |  |  |  | ✓ |
patchelf/0.10-GCCcore-8.3.0 |  |  | ✓ |  |
PCRE/8.38-foss-2016b |  |  |  | ✓ |
PCRE/8.38-intel-2016a |  |  |  | ✓ |
PCRE/8.39-foss-2016b |  |  |  | ✓ |
PCRE/8.39-foss-2018a |  |  |  | ✓ |
PCRE/8.39-intel-2016b |  |  |  | ✓ |
PCRE/8.41-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ | ✓
PCRE/8.41-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
PCRE/8.43-GCCcore-8.2.0 |  |  |  | ✓ |
PCRE/8.43-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
PCRE/8.44-GCCcore-10.2.0 |  | ✓ | ✓ |  | ✓
PCRE/8.44-GCCcore-9.3.0 |  | ✓ |  | ✓ |
PCRE2/10.33-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
PCRE2/10.34-GCCcore-9.3.0 |  | ✓ |  | ✓ |
PCRE2/10.35-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
PDT/3.25.1-GCCcore-9.3.0 | ✓ |  |  |  |
Perl/5.22.2-intel-2016a |  |  |  | ✓ |
Perl/5.24.0-foss-2016b |  |  |  | ✓ |
Perl/5.24.0-intel-2016b |  |  |  | ✓ |
Perl/5.24.0-intel-2018a |  |  |  | ✓ |
Perl/5.26.0-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ | ✓
Perl/5.26.0-intel-2017b |  |  |  | ✓ |
Perl/5.26.1-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ |
Perl/5.28.0-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
Perl/5.28.1-GCCcore-8.2.0 |  |  |  | ✓ |
Perl/5.30.0-GCCcore-8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
Perl/5.30.2-GCCcore-9.3.0 | ✓ | ✓ |  | ✓ |
Perl/5.32.0-GCCcore-10.2.0 | ✓ | ✓ | ✓ | ✓ | ✓
PETSc/3.11.0-foss-2018b-downloaded-deps |  |  |  | ✓ |
PETSc/3.12.4-foss-2019b |  |  |  | ✓ |
PETSc/3.7.2-intel-2016a-Python-2.7.11 |  |  |  | ✓ |
PGI/12.10 |  | ✓ |  |  |
PGI/16.5 |  | ✓ |  |  |
PGI/19.7-GCC-8.3.0-2.32 |  |  |  |  | ✓
phonopy/1.12.2.20-intel-2017b-Python-2.7.14 |  | ✓ |  |  |
Pillow-SIMD/6.0.x.post0-GCCcore-8.3.0 |  |  | ✓ |  |
Pillow/6.0.0-GCCcore-8.2.0 |  |  |  | ✓ |
Pillow/6.2.1-GCCcore-8.3.0 |  |  | ✓ | ✓ | ✓
plc/3.0.1-foss-2019a-Python-2.7.15 |  |  |  | ✓ |
plc/3.0.1-foss-2020a-Python-3.8.2 |  |  |  | ✓ |
plc/3.01-foss-2020a-Python-3.8.2 | ✓ |  |  |  |
PLUMED/2.4.0-intel-2017b |  | ✓ |  | ✓ |
PLUMED/2.4.0-intel-2018a |  |  | ✓ |  |
PLUMED/2.4.2-intel-2018b |  |  |  |  | ✓
PLUMED/2.5.3-foss-2019b-Python-3.7.4 |  | ✓ |  |  |
PLUMED/2.6.0-intel-2020a-Python-3.8.2 |  |  |  | ✓ |
PMIx/3.1.5-GCCcore-10.2.0 |  | ✓ | ✓ | ✓ | ✓
PMIx/3.1.5-GCCcore-9.3.0 |  | ✓ |  |  |
poetry/1.0.9-GCCcore-9.3.0-Python-3.8.2 |  | ✓ |  |  |
PostgreSQL/9.6.2-foss-2016b-Python-2.7.12 |  |  |  | ✓ |
POV-Ray/3.7.0.7-intel-2018b |  | ✓ |  |  |
PROJ/4.9.2-foss-2016b |  |  |  | ✓ |
PROJ/4.9.3-foss-2016b |  |  |  | ✓ |
PROJ/4.9.3-foss-2017b |  |  |  | ✓ | ✓
PROJ/4.9.3-foss-2018a |  |  |  | ✓ |
PROJ/4.9.3-intel-2016b |  |  |  | ✓ |
PROJ/5.0.0-foss-2018b |  | ✓ | ✓ | ✓ | ✓
PROJ/5.0.0-intel-2018b |  |  |  |  | ✓
PROJ/6.2.1-GCCcore-8.3.0 | ✓ |  | ✓ |  |
PROJ/7.0.0-GCCcore-9.3.0 | ✓ | ✓ |  | ✓ |
PROJ/7.2.1-GCCcore-10.2.0 |  |  | ✓ |  | ✓
protobuf-python/3.10.0-foss-2019b-Python-3.7.4 |  |  | ✓ |  | ✓
protobuf-python/3.10.0-fosscuda-2019b-Python-3.7.4 |  |  | ✓ |  | ✓
protobuf-python/3.14.0-GCCcore-10.2.0 | ✓ |  |  |  |
protobuf-python/3.4.0-intel-2017b-Python-3.6.3 |  |  |  | ✓ |
protobuf/3.10.0-GCCcore-8.3.0 | ✓ |  | ✓ | ✓ | ✓
protobuf/3.14.0-GCCcore-10.2.0 | ✓ |  |  |  |
protobuf/3.4.0-intel-2017b |  |  |  | ✓ |
protobuf/3.6.1-GCCcore-7.3.0 |  |  | ✓ | ✓ | ✓
psycopg2/2.7-foss-2016b-Python-2.7.12 |  |  |  | ✓ |
pybedtools/0.8.1-foss-2019b |  |  |  | ✓ |
pybind11/2.4.3-GCCcore-8.3.0-Python-3.7.4 |  | ✓ | ✓ | ✓ | ✓
pybind11/2.4.3-GCCcore-9.3.0-Python-3.8.2 |  | ✓ |  | ✓ |
pybind11/2.6.0-GCCcore-10.2.0 |  |  | ✓ |  | ✓
pybinding/0.9.4-foss-2018b-Python-3.6.6 |  | ✓ |  | ✓ |
PyCairo/1.18.0-foss-2018b-Python-2.7.15 |  |  |  |  | ✓
PyGObject/2.28.7-foss-2018b-Python-2.7.15 |  |  |  |  | ✓
PyGTK/2.24.0-foss-2018b-Python-2.7.15 |  |  |  |  | ✓
PyOpenGL/3.1.3b2-foss-2018b-Python-2.7.15 |  |  |  |  | ✓
pyproj/2.6.1.post1-GCCcore-9.3.0-Python-3.8.2 | ✓ |  |  |  |
PyQt/4.12-foss-2016b-Python-2.7.12 |  |  |  | ✓ |
PyQt5/5.7.1-foss-2016b-Python-2.7.12 |  |  |  | ✓ |
Pysam/0.15.1-foss-2018b-Python-3.6.6 |  | ✓ |  |  |
Pysam/0.15.3-GCC-8.3.0 |  |  |  | ✓ |
Pysam/0.16.0.1-GCC-9.3.0 |  | ✓ |  |  |
PyTables/3.6.1-foss-2019b-Python-3.7.4 |  | ✓ |  |  |
pytest/3.8.0-foss-2018a-Python-3.6.4 |  | ✓ | ✓ |  |
Python/2.7.11-intel-2016a |  |  |  | ✓ |
Python/2.7.12-foss-2016b |  |  |  | ✓ |
Python/2.7.14-foss-2017b |  |  | ✓ | ✓ | ✓
Python/2.7.14-foss-2018a |  | ✓ | ✓ | ✓ |
Python/2.7.14-GCCcore-6.4.0-bare |  | ✓ | ✓ | ✓ | ✓
Python/2.7.14-intel-2017b |  | ✓ | ✓ | ✓ | ✓
Python/2.7.14-intel-2018a |  |  | ✓ |  |
Python/2.7.15-foss-2018b |  | ✓ | ✓ | ✓ | ✓
Python/2.7.15-fosscuda-2018b |  |  | ✓ |  | ✓
Python/2.7.15-GCCcore-7.3.0-bare |  | ✓ | ✓ | ✓ | ✓
Python/2.7.15-GCCcore-8.2.0 |  |  |  | ✓ |
Python/2.7.15-intel-2018b |  | ✓ | ✓ | ✓ | ✓
Python/2.7.16-GCCcore-8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
Python/2.7.18-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
Python/2.7.18-GCCcore-9.3.0 | ✓ | ✓ |  | ✓ |
Python/3.5.1-intel-2016a |  |  |  | ✓ |
Python/3.5.2-foss-2016b |  |  |  | ✓ |
Python/3.6.3-foss-2017b |  |  |  | ✓ | ✓
Python/3.6.3-foss-2018a |  |  |  | ✓ |
Python/3.6.3-fosscuda-2017b |  |  |  |  | ✓
Python/3.6.3-intel-2017b |  |  |  | ✓ | ✓
Python/3.6.4-foss-2018a |  | ✓ | ✓ | ✓ |
Python/3.6.4-fosscuda-2018a |  |  | ✓ |  |
Python/3.6.4-intel-2018a |  |  | ✓ | ✓ |
Python/3.6.6-foss-2018b |  | ✓ | ✓ | ✓ | ✓
Python/3.6.6-fosscuda-2018b |  |  | ✓ |  | ✓
Python/3.6.6-intel-2018b |  | ✓ | ✓ |  | ✓
Python/3.7.0-foss-2018b |  |  |  |  | ✓
Python/3.7.2-GCCcore-8.2.0 |  |  |  | ✓ |
Python/3.7.4-GCCcore-8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
Python/3.8.2-GCCcore-9.3.0 | ✓ | ✓ |  | ✓ |
Python/3.8.6-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
PyTorch/1.4.0-foss-2019b-Python-3.7.4 |  |  | ✓ | ✓ |
PyTorch/1.4.0-fosscuda-2019b-Python-3.7.4 |  |  | ✓ |  |
PyTorch/1.6.0-foss-2019b-Python-3.7.4 |  |  |  |  | ✓
PyTorch/1.6.0-fosscuda-2019b-Python-3.7.4 |  |  |  |  | ✓
PyYAML/3.12-intel-2017b-Python-2.7.14 |  | ✓ |  |  |
PyYAML/3.13-foss-2018b-Python-3.6.6 |  |  |  | ✓ |
PyYAML/3.13-fosscuda-2018b-Python-3.6.6 |  |  | ✓ |  |
PyYAML/5.1.2-GCCcore-8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
PyYAML/5.3-GCCcore-9.3.0 | ✓ |  |  |  |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
Q-Chem/5.1.2-foss-2018b |  | ✓ |  |  |
Q-Chem/5.2.1-intel-2018b |  | ✓ |  |  |
QCA/2.1.3-foss-2016b |  |  |  | ✓ |
QGIS/2.18.4-foss-2016b-Python-2.7.12 |  |  |  | ✓ |
Qhull/2015.2-foss-2016b |  |  |  | ✓ |
Qhull/2015.2-foss-2017b |  |  |  | ✓ |
Qhull/2015.2-foss-2018a |  |  |  | ✓ |
Qhull/2015.2-GCCcore-6.4.0 |  |  | ✓ | ✓ |
Qhull/2015.2-GCCcore-7.3.0 |  |  | ✓ | ✓ |
Qhull/2019.1-GCCcore-8.3.0 |  |  | ✓ |  | ✓
QIIME2/2019.7 |  |  |  | ✓ |
QJson/0.9.0-foss-2016b |  |  |  | ✓ |
qrupdate/1.1.2-foss-2018a |  |  | ✓ | ✓ |
qrupdate/1.1.2-foss-2018b |  |  | ✓ | ✓ |
qrupdate/1.1.2-GCCcore-8.3.0 |  |  | ✓ |  | ✓
QScintilla/2.10-foss-2016b-Python-2.7.12 |  |  |  | ✓ |
Qt/4.8.7-foss-2016b |  |  |  | ✓ |
Qt/4.8.7-foss-2017b |  |  |  | ✓ |
Qt/4.8.7-intel-2016a |  |  |  | ✓ |
Qt/4.8.7-intel-2017b |  |  |  | ✓ |
Qt5/5.10.1-foss-2018a |  | ✓ | ✓ | ✓ |
Qt5/5.10.1-foss-2018b |  | ✓ | ✓ | ✓ | ✓
Qt5/5.13.1-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
Qt5/5.14.1-GCCcore-9.3.0 | ✓ |  |  | ✓ |
Qt5/5.14.2-GCCcore-10.2.0 | ✓ |  |  |  |
Qt5/5.7.0-foss-2016b |  |  |  | ✓ |
QuantumESPRESSO/6.3-foss-2018b |  |  |  | ✓ |
QuantumESPRESSO/6.3-intel-2018b |  |  | ✓ | ✓ | ✓
QuantumESPRESSO/6.6-foss-2020b | ✓ |  |  |  |
QuantumESPRESSO/6.6-intel-2019b |  |  | ✓ |  | ✓
QuantumESPRESSO/6.7-foss-2019b | ✓ |  |  |  |
Qwt/6.1.3-foss-2016b |  |  |  | ✓ |
Qwt/6.1.4-GCCcore-8.3.0 |  | ✓ |  | ✓ |
QwtPolar/1.1.1-foss-2016b |  |  |  | ✓ |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
R-bundle-Bioconductor/3.11-foss-2020a-R-4.0.0 | ✓ | ✓ |  | ✓ |
R-bundle-Bioconductor/3.12-foss-2020b-R-4.0.3 | ✓ | ✓ |  |  |
R/3.4.1-foss-2016b-X11-20160819 |  |  |  | ✓ |
R/3.4.3-foss-2017b-X11-20171023 |  |  |  | ✓ | ✓
R/3.5.1-foss-2018b |  | ✓ | ✓ | ✓ |
R/3.6.2-foss-2019b | ✓ | ✓ | ✓ | ✓ | ✓
R/4.0.0-foss-2020a | ✓ | ✓ |  | ✓ |
R/4.0.3-foss-2020b | ✓ | ✓ |  |  |
R/4.0.4-foss-2020b |  |  | ✓ |  |
rainbow/2.0.4-foss-2019b |  |  |  | ✓ |
RapidJSON/1.1.0-GCCcore-8.3.0 |  | ✓ |  |  |
RAxML-NG/0.9.0-gompi-2019b |  |  |  | ✓ |
RAxML/8.2.12-iimpi-2019b-hybrid-avx2 |  |  |  | ✓ |
re2c/1.2.1-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
re2c/1.3-GCCcore-9.3.0 | ✓ |  |  | ✓ |
re2c/2.0.3-GCCcore-10.2.0 | ✓ |  |  |  |
RELION/3.0.4-foss-2017b |  |  | ✓ |  |
RELION/3.0.4-intel-2017b |  |  |  |  | ✓
RELION/3.1.1-foss-2019b |  |  | ✓ |  |
RELION/3.1.1-fosscuda-2019b |  |  | ✓ |  |
RepastHPC/2.2.0-foss-2018a |  |  |  | ✓ |
requests/2.13.0-foss-2016b-Python-2.7.12 |  |  |  | ✓ |
rgdal/1.4-8-foss-2020a-R-4.0.0 |  | ✓ |  | ✓ |
rjags/4-10-foss-2020a-R-4.0.0 |  | ✓ |  | ✓ |
RNAmmer/1.2-foss-2018b-Perl-5.28.0-HMMER-2.3.2 |  | ✓ |  |  |
rnaQUAST/1.5.2-foss-2018b-Python-2.7.15 |  | ✓ |  | ✓ |
ROOT/6.20.04-foss-2019b-Python-3.7.4 | ✓ |  | ✓ | ✓ |
ROOT/6.24.00-foss-2020b-Python-3.8.6 | ✓ |  |  |  |
Rosetta/2016.28.58794-foss-2016b |  | ✓ |  |  |
Rosetta/3.7-foss-2016b |  | ✓ |  |  |
RSEM/1.3.3-foss-2019b |  |  |  | ✓ |
RStudio-Server/1.2.5042-foss-2019b-Java-11 |  |  |  | ✓ |
Ruby/2.3.1-intel-2016b |  |  |  | ✓ |
Ruby/2.3.1-intel-2018a |  |  |  | ✓ |
Ruby/2.5.0-intel-2017b |  |  |  | ✓ |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
Salmon/0.14.1-foss-2018b |  | ✓ |  | ✓ |
Salmon/0.8.2 |  |  |  | ✓ |
Salmon/1.3.0-gompi-2020a |  |  |  | ✓ |
SAMtools/1.10-GCC-8.3.0 | ✓ |  |  | ✓ |
SAMtools/1.10-GCC-9.3.0 | ✓ |  |  |  |
SAMtools/1.11-GCC-9.3.0 |  | ✓ |  |  |
SAMtools/1.3.1-foss-2018a |  |  |  | ✓ |
SAMtools/1.5-foss-2016b |  |  |  | ✓ |
SAMtools/1.6-foss-2018a |  |  |  | ✓ |
SAMtools/1.9-foss-2018b |  | ✓ |  | ✓ | ✓
SAMtools/1.9-intel-2018b |  |  |  | ✓ |
ScaLAPACK/2.0.2-gompi-2016b-OpenBLAS-0.2.18-LAPACK-3.6.1 |  |  |  | ✓ | ✓
ScaLAPACK/2.0.2-gompi-2017b-OpenBLAS-0.2.20 |  | ✓ | ✓ | ✓ | ✓
ScaLAPACK/2.0.2-gompi-2018a-OpenBLAS-0.2.20 |  | ✓ | ✓ | ✓ |
ScaLAPACK/2.0.2-gompi-2018b-OpenBLAS-0.3.1 |  | ✓ | ✓ | ✓ | ✓
ScaLAPACK/2.0.2-gompi-2019a-OpenBLAS-0.3.5 |  |  |  | ✓ |
ScaLAPACK/2.0.2-gompi-2019b | ✓ | ✓ | ✓ | ✓ | ✓
ScaLAPACK/2.0.2-gompic-2017b-OpenBLAS-0.2.20 |  |  |  |  | ✓
ScaLAPACK/2.0.2-gompic-2018a-OpenBLAS-0.2.20 |  |  | ✓ |  |
ScaLAPACK/2.0.2-gompic-2018b-OpenBLAS-0.3.1 |  | ✓ | ✓ |  | ✓
ScaLAPACK/2.0.2-gompic-2019b |  | ✓ | ✓ |  | ✓
ScaLAPACK/2.1.0-gompi-2020a | ✓ | ✓ |  | ✓ |
ScaLAPACK/2.1.0-gompi-2020a-bf | ✓ |  |  |  |
ScaLAPACK/2.1.0-gompi-2020b | ✓ | ✓ | ✓ | ✓ | ✓
ScaLAPACK/2.1.0-gompic-2020a |  | ✓ |  |  |
Scalasca/2.5-gompi-2020a | ✓ |  |  |  |
Schrodinger/2019-1_Linux-x86_64 |  | ✓ |  |  |
ScientificPython/2.9.4-intel-2016a-Python-2.7.11 |  |  |  | ✓ |
scikit-build/0.10.0-foss-2020a-Python-3.8.2 | ✓ |  |  |  |
scikit-build/0.10.0-fosscuda-2020a-Python-3.8.2 |  | ✓ |  |  |
scikit-learn/0.20.0-fosscuda-2018b-Python-3.6.6 |  |  | ✓ |  |
scikit-learn/0.20.2-foss-2018b-Python-3.6.6 |  |  |  | ✓ |
scikit-learn/0.21.3-foss-2019b-Python-3.7.4 | ✓ | ✓ |  | ✓ |
scikit-learn/0.22.1-fosscuda-2019b-Python-3.7.4 |  |  | ✓ |  |
scikit-learn/0.23.1-foss-2020a-Python-3.8.2 | ✓ | ✓ |  |  |
scikit-learn/0.23.2-foss-2020b | ✓ |  |  |  |
scip/5.0.1-foss-2017b |  |  |  | ✓ |
SciPy-bundle/2019.03-foss-2019a |  |  |  | ✓ |
SciPy-bundle/2019.10-foss-2019b-Python-2.7.16 |  | ✓ |  | ✓ | ✓
SciPy-bundle/2019.10-foss-2019b-Python-3.7.4 | ✓ | ✓ | ✓ | ✓ | ✓
SciPy-bundle/2019.10-fosscuda-2019b-Python-3.7.4 |  | ✓ | ✓ |  | ✓
SciPy-bundle/2019.10-intel-2019b-Python-2.7.16 |  |  |  | ✓ |
SciPy-bundle/2019.10-intel-2019b-Python-3.7.4 |  |  |  | ✓ |
SciPy-bundle/2020.03-foss-2020a-Python-2.7.18 | ✓ |  |  |  |
SciPy-bundle/2020.03-foss-2020a-Python-3.8.2 | ✓ | ✓ |  | ✓ |
SciPy-bundle/2020.03-fosscuda-2020a-Python-3.8.2 |  | ✓ |  |  |
SciPy-bundle/2020.03-intel-2020a-Python-3.8.2 |  |  |  | ✓ |
SciPy-bundle/2020.11-foss-2020b | ✓ |  | ✓ |  | ✓
scipy/1.4.1-foss-2019b-Python-3.7.4 |  | ✓ |  |  |
scipy/1.4.1-fosscuda-2019b-Python-3.7.4 |  |  |  |  | ✓
SCons/3.1.1-GCCcore-8.3.0 |  | ✓ |  | ✓ |
Score-P/6.0-gompi-2020a | ✓ |  |  |  |
SCOTCH/5.1.12b_esmumps-foss-2017b |  |  |  | ✓ |
SCOTCH/6.0.4-foss-2016b |  |  |  | ✓ |
SCOTCH/6.0.4-foss-2017b |  |  |  | ✓ |
SCOTCH/6.0.4-foss-2018a |  |  |  | ✓ |
SCOTCH/6.0.4-intel-2016a |  |  |  | ✓ |
SCOTCH/6.0.4-intel-2017b |  |  |  | ✓ |
SCOTCH/6.0.5-foss-2018b |  |  |  | ✓ |
SCOTCH/6.0.6-foss-2018b |  |  | ✓ | ✓ | ✓
SCOTCH/6.0.9-gompi-2019b |  |  | ✓ | ✓ | ✓
SCOTCH/6.0.9-gompi-2020a | ✓ |  |  | ✓ |
SCOTCH/6.0.9-iimpi-2019b |  |  |  | ✓ |
SCOTCH/6.1.0-gompi-2020b | ✓ |  |  |  |
scVelo/0.1.24-foss-2019b-Python-3.7.4 |  | ✓ |  |  |
SDL2/2.0.9-intel-2018b |  | ✓ |  |  |
SHAPEIT4/4.1.3-foss-2019b | ✓ | ✓ | ✓ | ✓ | ✓
SHAPEIT4/4.2.0-foss-2020b | ✓ | ✓ | ✓ | ✓ | ✓
SHAPEIT4/4.2.1-foss-2020b |  | ✓ | ✓ |  |
Shapely/1.7.1-GCC-9.3.0-Python-3.8.2 | ✓ |  |  |  |
Siesta/4.1-b4-intel-2018b |  |  | ✓ |  |
SignalP/4.1g-foss-2018b-Perl-5.28.0 |  | ✓ |  |  |
Singularity/2.5.2-foss-2017b |  |  |  | ✓ |
SIONlib/1.7.6-GCCcore-9.3.0-tools | ✓ |  |  |  |
SIP/4.19-foss-2016b-Python-2.7.12 |  |  |  | ✓ |
SLEPc/3.12.2-foss-2019b |  |  |  | ✓ |
snappy/1.1.7-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
snappy/1.1.8-GCCcore-10.2.0 | ✓ |  |  |  |
snappy/1.1.8-GCCcore-9.3.0 | ✓ |  |  | ✓ |
SPAdes/3.13.0-foss-2018b |  | ✓ |  | ✓ |
sparsehash/2.0.2-foss-2018a |  |  |  | ✓ |
sparsehash/2.0.3-GCCcore-6.4.0 |  |  |  |  | ✓
sparsehash/2.0.3-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
sparsehash/2.0.3-GCCcore-8.3.0 |  | ✓ |  |  |
sparsehash/2.0.4-GCCcore-10.2.0 |  | ✓ |  |  |
Sphinx/1.8.1-foss-2018b-Python-2.7.15 |  | ✓ |  |  |
SQLite/3.13.0-foss-2016b |  |  |  | ✓ |
SQLite/3.13.0-foss-2018a |  |  |  | ✓ |
SQLite/3.20.1-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ | ✓
SQLite/3.21.0-GCCcore-6.4.0 |  | ✓ | ✓ | ✓ |
SQLite/3.24.0-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
SQLite/3.27.2-GCCcore-8.2.0 |  |  |  | ✓ |
SQLite/3.29.0-GCCcore-8.3.0 | ✓ | ✓ | ✓ | ✓ | ✓
SQLite/3.31.1-GCCcore-9.3.0 | ✓ | ✓ |  | ✓ |
SQLite/3.33.0-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
SQLite/3.9.2-intel-2016a |  |  |  | ✓ |
Stacks/1.47-foss-2018a |  |  |  | ✓ |
Stacks/1.48-intel-2018b |  |  |  | ✓ |
Stacks/2.41-foss-2018b |  |  | ✓ | ✓ |
STAR/2.7.3a-GCC-8.3.0 | ✓ | ✓ |  | ✓ |
statsmodels/0.11.0-foss-2019b-Python-3.7.4 |  | ✓ |  |  |
statsmodels/0.11.1-foss-2020a-Python-3.8.2 | ✓ |  |  |  |
stda/1.6-1-intel-2017b |  | ✓ |  |  |
stda/1.6-intel-2017b |  | ✓ |  |  |
stda/1.6.2-intel-2020b-64 |  | ✓ |  |  |
StringTie/1.3.5-foss-2018b |  |  |  | ✓ |
Subversion/1.14.0-GCCcore-10.2.0 | ✓ |  |  |  |
Subversion/1.14.0-GCCcore-9.3.0 | ✓ |  |  |  |
SuiteSparse/4.5.3-intel-2016a-ParMETIS-4.0.3 |  |  |  | ✓ |
SuiteSparse/5.1.2-foss-2018a-METIS-5.1.0 |  |  | ✓ | ✓ |
SuiteSparse/5.1.2-foss-2018b-METIS-5.1.0 |  | ✓ | ✓ | ✓ |
SuiteSparse/5.1.2-intel-2018b-METIS-5.1.0 |  |  |  | ✓ |
SuiteSparse/5.6.0-foss-2019b-METIS-5.1.0 |  |  | ✓ | ✓ | ✓
swak4Foam/20190423-foss-2018b-OpenFOAM-v1806 |  |  |  | ✓ |
SWIG/3.0.11-foss-2016b-Python-2.7.12 |  |  |  | ✓ |
SWIG/3.0.12-foss-2017b-Python-2.7.14 |  |  |  | ✓ | ✓
SWIG/3.0.12-foss-2017b-Python-3.6.3 |  |  |  | ✓ |
SWIG/3.0.12-foss-2018a-Python-3.6.3 |  |  |  | ✓ |
SWIG/3.0.12-foss-2018a-Python-3.6.4 |  |  | ✓ |  |
SWIG/3.0.12-foss-2018b-Python-3.6.6 |  | ✓ | ✓ | ✓ | ✓
SWIG/3.0.12-intel-2017b-Python-2.7.14 |  |  | ✓ |  |
SWIG/3.0.12-intel-2017b-Python-3.6.3 |  |  |  | ✓ |
SWIG/3.0.12-intel-2018b-Python-3.6.6 |  |  |  |  | ✓
SWIG/3.0.8-intel-2016a-Python-2.7.11 |  |  |  | ✓ |
SWIG/4.0.1-GCCcore-8.3.0 | ✓ | ✓ | ✓ |  | ✓
SWIG/4.0.1-GCCcore-9.3.0 | ✓ | ✓ |  | ✓ |
SWIG/4.0.2-GCCcore-10.2.0 | ✓ |  |  |  |
sympy/1.0-intel-2016a-Python-2.7.11 |  |  |  | ✓ |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
T-REX/3.0.2-PGI-16.10-CDK-GCC-5.4.0-2.26 |  | ✓ |  |  |
tabix/0.2.6-GCCcore-7.3.0 |  | ✓ |  | ✓ |
TAU/2.30-gompi-2020a | ✓ |  |  |  |
tbb/2018_U5-GCCcore-6.4.0 |  |  | ✓ |  | ✓
tbb/2018_U5-GCCcore-7.3.0 |  | ✓ | ✓ | ✓ | ✓
tbb/2019_U9-GCCcore-8.3.0 |  | ✓ |  |  | ✓
tbb/2020.1-GCCcore-9.3.0 |  |  |  | ✓ |
tbb/2020.2-GCCcore-8.3.0 |  |  | ✓ |  |
TensorFlow/1.12.0-foss-2018b-Python-3.6.6 |  |  | ✓ | ✓ |
TensorFlow/1.12.0-fosscuda-2018b-Python-3.6.6 |  |  | ✓ |  |
TensorFlow/1.15.2-foss-2019b-Python-3.7.4 | ✓ |  |  | ✓ |
TensorFlow/2.0.0-fosscuda-2019b-Python-3.7.4 |  |  | ✓ |  | ✓
TensorFlow/2.2.0-foss-2019b-Python-3.7.4 |  |  |  | ✓ |
TensorFlow/2.3.1-fosscuda-2019b-Python-3.7.4 |  |  | ✓ |  | ✓
TensorFlow/2.4.1-foss-2020b | ✓ |  |  |  |
texinfo/6.5-GCCcore-6.4.0 |  |  | ✓ | ✓ |
texinfo/6.5-GCCcore-7.3.0 |  |  | ✓ | ✓ |
texinfo/6.7-GCCcore-8.3.0 |  |  | ✓ |  | ✓
Theano/1.0.3-foss-2018b-Python-3.6.6 |  |  |  | ✓ |
Theano/1.0.4-fosscuda-2018b-Python-3.6.6 |  |  | ✓ |  |
Theano/1.0.4-fosscuda-2019b-Python-3.7.4 |  |  |  |  | ✓
tidybayes/2.1.1-foss-2020a-R-4.0.0 |  | ✓ |  |  |
tidymodels/0.1.0-foss-2020a-R-4.0.0 |  | ✓ |  | ✓ |
TINKER/8.6.1-foss-2018b |  |  | ✓ |  | ✓
TINKER/8.7.2-foss-2019b |  |  | ✓ |  | ✓
TMHMM/2.0c-foss-2018b-Perl-5.28.0 |  | ✓ |  |  |
TmoleX/4.5 |  | ✓ |  |  |
Togl/2.0-GCCcore-8.3.0 |  |  | ✓ |  | ✓
torchvision/0.5.0-fosscuda-2019b-Python-3.7.4-PyTorch-1.4.0 |  |  | ✓ |  |
tqdm/4.41.1-GCCcore-8.3.0 |  | ✓ |  |  |
tqdm/4.47.0-GCCcore-9.3.0 | ✓ |  |  |  |
Trans-ABySS/2.0.1-foss-2018b-Python-3.6.6 |  | ✓ |  |  |
TransDecoder/5.5.0-foss-2018b-Perl-5.28.0 |  | ✓ |  |  |
Transrate/1.0.3 |  |  |  | ✓ |
Trimmomatic/0.38-Java-1.8 |  |  |  | ✓ |
Trinity/2.8.4-foss-2018b |  | ✓ |  | ✓ |
Trinotate/3.2.0-foss-2018b-Perl-5.28.0 |  | ✓ |  |  |
TURBOMOLE/6.0.2 |  | ✓ |  |  |
TURBOMOLE/7.2.1 |  | ✓ |  |  |
TURBOMOLE/7.3.1 |  | ✓ |  |  |
TURBOMOLE/7.4.1 |  | ✓ |  |  |
TURBOMOLE/7.5 |  | ✓ |  |  |
typing-extensions/3.7.4.3-GCCcore-10.2.0 | ✓ |  |  |  |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
UCX/1.8.0-GCCcore-9.3.0 |  | ✓ |  | ✓ |
UCX/1.8.0-GCCcore-9.3.0-CUDA-11.0.2 |  | ✓ |  |  |
UCX/1.9.0-GCCcore-10.2.0 |  | ✓ | ✓ | ✓ | ✓
UDUNITS/1.12.11-iimpi-2019b |  |  |  | ✓ |
UDUNITS/2.2.24-intel-2018a |  |  |  | ✓ |
UDUNITS/2.2.25-intel-2017b |  |  | ✓ | ✓ |
UDUNITS/2.2.26-foss-2018b |  | ✓ | ✓ | ✓ | ✓
UDUNITS/2.2.26-foss-2020a |  | ✓ |  |  |
UDUNITS/2.2.26-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
UDUNITS/2.2.26-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
UDUNITS/2.2.26-GCCcore-9.3.0 |  |  |  | ✓ |
UDUNITS/2.2.26-intel-2016a |  |  |  | ✓ |
UDUNITS/2.2.26-intel-2017b |  |  | ✓ | ✓ |
UDUNITS/2.2.26-intel-2018b |  |  | ✓ | ✓ | ✓
UnZip/6.0-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
UnZip/6.0-GCCcore-9.3.0 | ✓ | ✓ |  |  |
USPEX/9.4.4-foss-2016b-Python-2.7.12 |  | ✓ |  |  |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
Valgrind/3.14.0-foss-2018b |  |  |  | ✓ |
Valgrind/3.16.1-gompi-2019b | ✓ |  |  |  |
Valgrind/3.16.1-gompi-2020a | ✓ |  |  |  |
Valgrind/3.16.1-iimpi-2020a | ✓ |  |  |  |
VASP/5.4.4-18Apr17-p01-intel-2020b | ✓ |  |  |  |
VASP/5.4.4-intel-2016b |  | ✓ |  |  |
VASP/5.4.4-intel-2019b |  |  |  | ✓ |
VCFtools/0.1.16-GCC-8.3.0 |  | ✓ |  |  |
VCFtools/0.1.16-GCC-9.3.0 | ✓ |  |  |  |
velocyto/0.17.17-foss-2020a-Python-3.8.2 |  | ✓ |  |  |
VirSorter/20160601-foss-2019b-Perl-5.30.0 |  |  |  | ✓ |
VMD/1.9.3-intel-2018b-Python-2.7.15 |  | ✓ | ✓ |  | ✓
Voro++/0.4.6-foss-2019b |  | ✓ |  |  |
VSEARCH/2.9.1-foss-2019b |  |  |  | ✓ |
VTK/8.2.0-foss-2019b-Python-3.7.4 |  | ✓ |  | ✓ |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
Wannier90/2.0.1.1-intel-2018b-abinit |  | ✓ |  | ✓ |
Wannier90/3.1.0-intel-2019b |  |  | ✓ |  | ✓
WannierTools/2.3.0-intel-2019b |  |  | ✓ |  | ✓
WannierTools/2.4.1-intel-2019b |  |  | ✓ |  | ✓
wheel/0.30.0-fosscuda-2017b-Python-3.6.3 |  |  |  |  | ✓
wheel/0.31.1-foss-2018b-Python-2.7.15 |  | ✓ |  |  |
wheel/0.31.1-foss-2018b-Python-3.6.6 |  |  | ✓ | ✓ | ✓
wheel/0.31.1-fosscuda-2018b-Python-2.7.15 |  |  |  |  | ✓
wheel/0.31.1-fosscuda-2018b-Python-3.6.6 |  |  | ✓ |  | ✓

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
x264/20160430-intel-2016a |  |  |  | ✓ |
x264/20181203-GCCcore-7.3.0 |  | ✓ | ✓ |  | ✓
x264/20190925-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
x264/20191217-GCCcore-9.3.0 | ✓ |  |  | ✓ |
x264/20201026-GCCcore-10.2.0 | ✓ |  |  |  |
x265/2.9-GCCcore-7.3.0 |  | ✓ | ✓ |  | ✓
x265/3.2-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
x265/3.3-GCCcore-10.2.0 | ✓ |  |  |  |
x265/3.3-GCCcore-9.3.0 | ✓ |  |  | ✓ |
XCrySDen/1.6.2-foss-2019b |  |  | ✓ |  | ✓
Xerces-C++/3.2.2-GCCcore-8.3.0 |  |  | ✓ |  |
Xerces-C++/3.2.3-GCCcore-10.2.0 | ✓ |  |  |  |
XML-LibXML/2.0132-GCCcore-7.3.0-Perl-5.28.0 |  | ✓ |  | ✓ |
XML-LibXML/2.0201-GCCcore-8.3.0 |  |  |  | ✓ |
XML-Parser/2.44_01-foss-2016b-Perl-5.24.0 |  |  |  | ✓ |
XML-Parser/2.44_01-GCCcore-6.4.0-Perl-5.26.0 |  |  | ✓ | ✓ | ✓
XML-Parser/2.44_01-GCCcore-6.4.0-Perl-5.26.1 |  |  | ✓ | ✓ |
XML-Parser/2.44_01-GCCcore-7.3.0-Perl-5.28.0 |  |  | ✓ | ✓ | ✓
xtb/5.8-intel-2017b |  | ✓ |  |  |
xtb/6.1b |  | ✓ |  |  |
xtb/6.2.3-foss-2019b |  |  | ✓ |  | ✓
xtb4stda/1.0-intel-2020b-64 |  | ✓ |  |  |
Xvfb/1.20.9-GCCcore-10.2.0 | ✓ | ✓ | ✓ |  | ✓
Xvfb/1.20.9-GCCcore-9.3.0 | ✓ | ✓ |  |  |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
yaff/1.6.0-foss-2019b-Python-3.7.4 |  | ✓ |  |  |
Yasm/1.3.0-GCCcore-10.2.0 | ✓ |  |  |  |
Yasm/1.3.0-GCCcore-7.3.0 |  | ✓ | ✓ |  | ✓
Yasm/1.3.0-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
Yasm/1.3.0-GCCcore-9.3.0 | ✓ |  |  | ✓ |
Yasm/1.3.0-intel-2016a |  |  |  | ✓ |
YAXT/0.5.1-intel-2016a |  |  |  | ✓ |
YAXT/0.5.1-intel-2016b |  |  |  | ✓ |
YAXT/0.5.1-intel-2017b |  |  |  | ✓ |
YAXT/0.5.1-intel-2018a |  |  |  | ✓ |
YAXT/0.6.2-iimpi-2019b |  |  |  | ✓ |
YAXT/0.9.0-gompi-2020b | ✓ |  |  |  |

Module | NIC5 | Hercules2 | Dragon2 | Lemaitre3 | Dragon1
ZeroMQ/4.2.5-foss-2018b |  | ✓ | ✓ |  | ✓
ZeroMQ/4.2.5-fosscuda-2018b |  |  | ✓ |  |
ZeroMQ/4.2.5-intel-2018b |  |  | ✓ |  | ✓
ZeroMQ/4.3.2-GCCcore-8.3.0 |  | ✓ | ✓ | ✓ | ✓
Zip/3.0-GCCcore-10.2.0 | ✓ |  |  |  |
Zip/3.0-GCCcore-8.3.0 | ✓ |  | ✓ | ✓ | ✓
zstd/1.4.4-GCCcore-9.3.0 |  | ✓ |  | ✓ |
zstd/1.4.5-GCCcore-10.2.0 |  | ✓ | ✓ |  | ✓

Last updated: 04 Jun 2021

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Software installed in the clusters »

Software installed classified by categories

If you follow the link for a given software in the list below, you’ll see a page with a minimal description followed by a table showing the modules available on each cluster for it.

The naming convention for each module follows the format software/version-toolchain, i.e. the module ABINIT/8.6.3-intel-2018a corresponds to the ABINIT software version 8.6.3 compiled with the intel-2018a toolchain. See here for an explanation about toolchains.

Biology

BioPerl

MrBayes

RePastHPC

Chemistry

OpenBabel

GROMACS

Atomistix ToolKit

NAMD

NWchem

ORCA

Rosetta

Compilers

CUDA

GNU GCC/Gfortran

icc

ifort

PGI

Data processing

HDF5 - file format and library for storing scientific data

Mathematics

Octave

Numerical libraries

arpack-ng

FFTW

GSL

Intel MKL

OpenBLAS

ScaLAPACK

Physics

ABINIT

DDSCAT

Meep

MPB

QuantumESPRESSO

Programming Languages

Perl

Python

R

Toolchains

foss

GCCcore

intel

Visualisation

gnuplot

Grace

Molden

ParaView

VMD

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Compiling and installing software from source

You will most probably, at some point, face a situation where you need additional software installed. One option is to ask the System Administrator to install the software globally on the cluster, which makes sense mainly for popular software used by many users.

Another option is to install the software locally in your home directory. Most of the time, this requires installing the software from the sources. If you do not have access to the sources, as is the case with many commercial software packages, you will need to check the documentation of the installation procedure to know how to install the software in a custom directory.

The whole process

Installing from source has an additional benefit: you then can tailor the compilation process to the hardware of the clusters.

It is customary to create a directory name .local in the home directory, to host the self-installed software. You can create it with

$ mkdir -p ~/.local/{bin,lib,src,include}

1. Download or upload the source code archive to your home directory on the cluster.

To do that, use the wget or the curl command. For instance, if the web link to the archive is https://www.repository.com/project/soft-2.3.tar.gz, issue the following command:

$ wget https://www.repository.com/project/soft-2.3.tar.gz

Alternatively, you can download the archive on your personal computer and then upload it on the cluster.

2. Unpack the archive

First go to the destination directory, for instance

$ cd ~/.local/src

and unpack the archive. If it is a .tar.gz or .tgz, simply run

$ tar -xzf soft-2.3.tar.gz

You might also encounter other archive types, such as .zip. Refer to this Wikipedia page [https://en.wikipedia.org/wiki/List_of_archive_formats] to choose the right tool for unpacking.

Once the files are extracted, go to the newly-created directory:

$ cd soft-2.3

3. Load compiler suite

Before compiling your code, you need to choose the compiler toolchain that suit your needs. You can check the different versions of foss or intel toolchains

For exmaple for the GNU Compiler Collection

$ module spider foss

------------------------------------------------------------------------------------------------------------------------------
  foss:
------------------------------------------------------------------------------------------------------------------------------
    Description:
      GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK
      support), FFTW and ScaLAPACK.

     Versions:
        foss/2014b
        foss/2015a
        foss/2015b
        ...

Or the intel compiler toolchain

$ ml spider intel

------------------------------------------------------------------------------------------------------------------------------
  intel:
------------------------------------------------------------------------------------------------------------------------------
    Description:
      Compiler toolchain including Intel compilers, Intel MPI and Intel Math Kernel Library (MKL).

     Versions:
        intel/2014b
        intel/2015a
        intel/2015b

Once you choose the toolchain you need to load it like:

$ module purge
The following modules were not unloaded:
  (Use "module --force purge" to unload all):

  1) databases   2) releases/2021b   3) tis/2017.01
$ module load releases/2021a intel/2021a

The following have been reloaded with a version change:
  1) releases/2021b => releases/2021a

The two module command must be later set in the submission script. This allow your executable to find the shared libraries in the compute nodes.

4. Run the ./configure script

First of all, read any README, or INSTALL file that you could find in the directory.

Then, run the ./configure script. That script will analyse the available software (compilers, GNU tools, etc.) that are needed for the compilation process, and prepare the subsequent compilation scripts.

At this stage, you will choose the directory to which the software must be installed. That is done with the –prefix option of the configure script. For example:

$ ./configure --prefix ~/.local/

You can also choose the compiler and compiler options with environment variables passed to ./configure, e.g. to use the Intel compiler: CC=icc ./configure. Other interesting variables include CFLAGS, CPPFLAGS, LDFLAGS. Run ./configure –help for a detailed list.

Check the output of the ./configure script; it may report missing dependencies, which often lead to deactivation of some features of the program.

4. Run make and make install

Once the ./configure script has run, you will be able to build the software with the make command. This step may take several minutes, depending on the complexity of the software. To speed things up, use the -j n option of make to run the build process in parallel on n CPUs. For instance:

$ make -j 4

The next step is to install the software in the destination directory.

$ make install

Now, you can optionally remove the source directory and the archive file to save disk space as the binaries will be copied to ~/.local/bin and the libraries will go to ~/.local/lib.

If you are following instructions on a web page and the instructions are to sudo make install, resist the urge to do so. Simply run make install.

Final steps

You will need to make sure your environment is properly configured to use the programs you just installed.

Once the software is properly installed, make sure to include the .local directories in the corresponding paths:

export PATH=~/.local/bin:$PATH
export LD_LIBRARY_PATH=~/.local/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=~/.local/lib:$LIBRARY_PATH
export CPATH=~/.local/include:$CPATH

It is advisable to put those lines in your .bash_profile file at your cluster home directory, which is sourced at each SSH login, to avoid typing them when starting each session.

To make sure your paths are correctly set, you can use the which command to see specifically which binary is called when you issue a given command, and ldd to see which dynamic libraries the binary is using.

Compiling for multiple CPU architectures

By default, compilers will tune the binary for the CPU of the machine they run on. So if you compile on, say cluster1 [https://www.ceci-hpc.be/clusters.html], which is equiped with Intel’s Westmere processors, and then run on the newer SandyBridge processors of cluster2 [https://www.ceci-hpc.be/clusters.html], the binaries of your program will not use the advanced features the SandyBridge processors offer. The two most common reasons for performances losses are reduced or inactive vectorisation (for instance not using the AVX SIMD instructions allow performing 8 double-precision floating point operations per clock cycle), and inefficient code scheduling (scheduling compute and I/O operations based on cache sizes and latencies that are not the correct ones).

What’s even worse, if you compile on a SandyBridge processor and then run on a Westmere processor, it might crash because it would be trying to use the AVX units which are absent on the latter processors.

With GCC

With GCC, you can mitigate this problem using the -march and -mtune arguments.

With the -march argument, you can prevent software crashes by telling the compiler to use only features that are present in earlier CPUs. If you specify -march=core2 then the resulting binary is guaranteed to work on every compute-node in the CÉCI clusters but with degraded performances in modern ones. If you specify -march=westmere, it will work everywhere except on Hmem (decommissioned). With -march=sandybridge, it will work only on Hercules, Dragon1. To target the AMD Epyc CPUs of Hercules and NIC5 you can specify -march=znver1 (Hercules) or -march=znver2 (NIC5). The later options require GCC version 9 and later.

The complete list of valid parameter values can be found in the GCC documentation [https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html].

With the -mtune argument, you can ask the compiler to optimize the binaries for a specific architecture, while remaining in the limits imposed by the -march option. It will mainly work on optimizing the instruction scheduling with respect to the CPU architecture. The -mtune argument accepts the same values as -march.

A safe option is thus to set -march to the CPU architecture of the oldest cluster you plan on using, and -mtune to the CPU architecture of the cluster you plan on using the most. The CPU architecture of each cluster can be found on the cluster page [https://www.ceci-hpc.be/clusters.html].

In your code, if you have functions that use CPU intrinsics or optimization features that are CPU specific, you can use a feature of GCC named Function multi-versioning [https://gcc.gnu.org/wiki/FunctionMultiVersioning]. The idea is to write several versions of the same function, with a specific __attribute__ to tell GCC which version goes for which CPU. Then, at runtime, the correct version of the function will be called based on the CPU on which it is running.

See the following example:

#include<iostream>
using namespace std;

__attribute__ ((target ("default")))
int foo ()
{
  // The default version of foo.
  return 0;
}

__attribute__ ((target ("sse4.2")))
int foo ()
{
  // foo version for SSE4.2
  return 1;
}

__attribute__ ((target ("avx")))
int foo ()
{
  // foo version for AVX
  return 2;
}

int main ()
{
  cout << "The function returned: " << foo() << "\n";
  return 0;
}

When compiled with g++ version 4.9 or above, it outputs 0 on Hmem, 1 on Lemaitre2 and 2 on Nic4.

With the Intel compiler

The Intel compiler has a very interesting feature called ‘Multiple code paths’. With the -x option, you can specify which CPU features the compiler is allowed to use, for instance -xSSE2 for CPUs that have the SSE2 feature, namely every current Intel or AMD CPU (beware that the -xSSE2 might lead to binaries that crash on AMD processors even if they have the SSE feature). If you specify -xSSE4.2, it will work everywhere except on Hmem. With -xAVX it will work only on Hercules, Dragon1, Vega and NIC4.

With the -ax option, you can specify additional so-called ‘code paths’ that the compiler will add to the binary, using additional sets of features. The ‘base path’ is specified with the -x parameter, and the -ax parameter allows compiling the code several times, each time for a different CPU architecture, and packing it all in a single binary. The Intel Compiler runtime will then decide, when the program runs, which portion of the binary to use based on the CPU of the machine it is running. So you could specify -axSSE4.2,AVX to build a binary optimized for all the CÉCI computers.

More information for these options are available on the Intel compiler website [https://software.intel.com/en-us/articles/performance-tools-for-software-developers-intel-compiler-options-for-sse-generation-and-processor-specific-optimizations], where the list of valid options is given. The option names are CPU features, rather than CPU architecture names as for GCC. The features a CPU offers can be found by looking at the contents of the /proc/cpuinfo file.

Note

The Intel compiler is developed and targeted for the Intel hardware and hence it has some minor issues when using it with AMD hardware. Execution of code compiled with the Intel compiler might lead to crash when executed on the AMD CPUs of Hercules and NIC5. The recommended flag to use the AVX2 instructions supported by the AMD EPYC CPUs is -march=core-avx2.

In some cases, when compiling with the -xavx, -xavx2 or -xcore-avx2 flags your application will abort with a message that the processor is not compatible:

Please verify that both the operating system and the proce_ssor support Intel(R) X87,
CMOV, MMX, FXSAVE, SSE, SSE2, SSE3, SSSE3, SSE4_1, SSE4_2, MOVBE, POPCNT, AVX, F16C, FMA,
BMI, LZCNT and AVX2 instructions.

To solve this problem, avoid using the above-mentioned flags for the compilation of the main() function. Avoiding the use of Interprocedural Optimization (remove the -ipo flag) may also be tried.

The Intel compiler has a feature similar to GCC’s function multiversioning, it is called Manual processor dispatch [https://software.intel.com/en-us/articles/how-to-manually-target-2nd-generation-intel-core-processors-with-support-for-intel-avx]

#include <stdio.h>

// need to create specific function versions for the following processors:
__declspec(cpu_dispatch(generic, core_i7_sse4_2, core_2nd_gen_avx))
// stub that will call the appropriate specific function version
int foo() {};

__declspec(cpu_specific(generic))
int foo() {
        return 0;
}

__declspec(cpu_specific(core_i7_sse4_2))
int foo() {
        return 1;
}

__declspec(cpu_specific(core_2nd_gen_avx))
int foo() {
        return 2;
}

int main() {
foo();
printf("Function returned: %d\n", foo());
return 0;
}

The above program, when compiled with the Intel compiler, will behave as the one with function multiversioning.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Installing software by yourself

Installing languages extensions

With scripting languages, even if the interpreter is installed globally, you have the possibility to install additional packages locally in your home directory.

Python

First of all, we discourage the use of Conda on the CÉCI clusters. See Conda .

For python include the --user option to pip install, e.g.:

$ pip install --user mymodule

will install the fictitious mymodule package in your home directory $HOME/.local/. Once that is done, you will need to make sure $HOME/.local/bin is in your PATH variable.

Make sure to replace the x.y part with the actual version of Python you are using. For instance:

$ export PATH=$HOME/.local/bin:$PATH
$ export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages/:$PYTHONPATH

It is important when you install a package that you load the correct Python module, and use the Pip option --no-binary :all: to recompile from source rather than install pre-compiled binaries whenever possible. See more information in the PIP documentation [https://pip.pypa.io/en/stable/reference/pip_install/#install-no-binary] . You can use GCC optimisation flags when doing so. Example:

CFLAGS='-O2 -pipe -march=sandybridge' pip install --no-binary :all: PACKAGE

The above example builds the PACKAGE with optimisation options that are compatible with most clusters, and hence suboptimal on recent ones. See With GCC for more information.

Virtualenvs

If you are already used to create Python virtualenvs for managing your custom modules installations (if you are not, is a good idea to learn about them [https://pythontips.com/2013/07/30/what-is-virtualenv]), take into account that on the clusters we provide, apart of different core Python versions, installations of different python modules bundles compatible with them. If you need a specific python module not available in the environment when you do e.g. module load Python/3.6.6-foss-2018b, check always with the module avail command or in the list of installed software if there is a specific installation provided for that Python installation.

In the case that you need mymodule which is not available, check carefully which are its dependencies and verify if some of those are available. Let’s imagine that mymodule has as requirements numpy, matplotlib, h5py and Keras. On some clusters there are specific installations for those, so you can load them

module load Python/3.6.6-foss-2018b
module load matplotlib/3.0.0-foss-2018b-Python-3.6.6
module load h5py/2.8.0-foss-2018b-Python-3.6.6
module load Keras/2.2.4-foss-2018b-Python-3.6.6

for numpy it is included already on the main Python module. Then you can proceed to create a virtualenv and install mymodule by doing:

mkdir ~/my_venv
virtualenv --system-site-packages ~/my_venv
source ~/my_venv/bin/activate
pip install mymodule

the --system-site-packages flag will make available inside the virtualenv the python modules already loaded. Then, when you pip install mymodule you avoid pulling the wheels for all those and you will be using the optimally compiled versions we provide.

R

If you run a script that depends on specific libraries/packages, those libraries need to be installed. A command like

library(doParallel)

will load the doParallel library if it is installed. If that library is not installed, then R will fail with the following explicit error mesage: Error in library(doParallel) : there is no package called 'doParallel'

If you write:

install.packages("doParallel")
library(doParallel)

then R will try to install the doParallel library before loading it.

On a HPC cluster, you will find that some libraries are installed beforehand by the system administrators. Note that libraries depends on the version of R so it could be that, on the same cluster, a library is available for one version and not for another version. For instance on Hmem, the doParallel library is not installed in R 2.13.1

ceciuser@cecicluster:~ $ R

R version 2.13.1 (2011-07-08)
[...]
> library(doParallel)
Error in library(doParallel) : there is no package called 'doParallel'
>

but it is in R 3.3.1

ceciuser@cecicluster:~ $ R

R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"
[...]
> library(doParallel)
Loading required package: foreach
foreach: simple, scalable parallel programming from Revolution Analytics
Use Revolution R for scalability, fault tolerance and more.
http://www.revolutionanalytics.com
Loading required package: iterators
Loading required package: parallel
>

Remember that you can choose the R version through environment modules and the module command.

If the library you need is not installed, you can either ask the system administrators to install it globally or install it by yourself for your account.

If you run the install.packages() command by yourself, R will note that you are not the administrator and ask whether it should create a private library where additional packages will be installed:

> install.packages('doParallel')
Warning in install.packages("doParallel") :
  'lib = "/opt/cecisw/arch/easybuild/2018b/software/R/3.5.1-foss-2018b/lib64/R/library"' is not writable   # This is because you do not have administrator access to the cluster
Would you like to use a personal library instead? (yes/No/cancel) yes
Would you like to create a personal library
‘~/R/x86_64-pc-linux-gnu-library/3.5’                                     # So R will create a directory in your home directory, one per R version.
to install packages into? (yes/No/cancel) yes
--- Please select a CRAN mirror for use in this session ---
Secure CRAN mirrors

 1: 0-Cloud [https]                   2: Algeria [https]
 3: Australia (Canberra) [https]      4: Australia (Melbourne 1) [https]
 5: Australia (Melbourne 2) [https]   6: Australia (Perth) [https]
 7: Austria [https]                   8: Belgium (Ghent) [https]
[...]
** testing if installed package can be loaded
* DONE (doParallel)

Once this is done, you will be able to load the library on all compute nodes of that cluster for the same R version as the one that was used to install the package.

Note that this process is interactive as R asks the user some questions, and this might fail when running inside a batch job. So you can run it interactively when connected to the frontend (the login node) once before you submit your jobs.

If you want to include the install.packages() command in your scripts, you should specify at least the target directory with the lib parameter and the mirror to use with the repo parameter:

install.packages('doParallel', lib='~/R/x86_64-pc-linux-gnu-library/3.5', repo='https://lib.ugent.be/CRAN')

so that R knows readily the answer to the questions.

As a final note, please be aware that on heterogeneous clusters (which have compute nodes with different generations of processors), the R installation is performed once per generation (to be optimised for the processors of that generation) and it could be that one library is missing for one R version on one generation of CPU.

Octave

With Octave you can use the pkg command [https://octave.org/doc/v6.4.0/Installing-and-Removing-Packages.html] to install additional packages, and the pkg prefix [http://www.gnu.org/software/octave/doc/interpreter/Administrating-Packages.html] command to decide where to install them. Here is an example how to install a package by yourself on the cluster. The objective of this mini tutorial is to have the azimuth function available on a specific version of Octave available as an system module. We will assume that the relevant system module is loaded.

First you need to find the name of the package which offers that function. According to its documentation, https://octave.sourceforge.io/mapping/function/azimuth.html, it is named “mapping”. The easy way would be using the Octave Forge [https://octave.sourceforge.io] from the Octave command line with

octave:1> pkg install -forge "mapping"

but unfortunately, in this example, the version available from the Octave Forge requires a newer version of Octave than we have currently active.

octave:1> pkg install -forge "mapping"
error: the following dependencies were unsatisfied:
   mapping needs octave >= 5.2.0
 mapping needs geometry >= 4.0.0
octave:2> version
ans = 4.4.1

We will then install the package and its dependencies manually. As the latest version is not working for this version, you will need to downloaded the previous version from the project page on Source Forge: https://sourceforge.net/projects/octave/files/Octave%20Forge%20Packages/Individual%20Package%20Releases/

Download the file mapping-1.4.0.tar.gz and copied it to your home directory.

Note

Whenever installing packages in Octave, load the texinfo module before starting Octave to be able to generate the documentation inside Octave.

As stated in the documentation, mapping requires some dependencies. When you try to install the package, with pkg install <path to file>, it complains:

octave:1> pkg install mapping-1.4.0.tar.gz
error: the following dependencies were unsatisfied:
   mapping needs geometry >= 4.0.0
octave:2> pkg install geometry-4.0.0.tar.gz
error: the following dependencies were unsatisfied:
   geometry needs matgeom >= 1.0.0

therefore, download the following files:

geometry-4.0.0.tar.gz

matgeom-1.2.3.tar.gz

from the same location and proceeded to install matgeom:

octave:1> pkg install matgeom-1.2.3.tar.gz
For information about changes from previous versions of the matgeom package, run 'news matgeom'.

and back the dependency chain:

octave:2> pkg install geometry-4.0.0.tar.gz
warning: doc_cache_create: unusable help text found in file 'clipper'
For information about changes from previous versions of the geometry package, run 'news geometry'.
octave:3> pkg install mapping-1.4.0.tar.gz
configure: WARNING: GDAL library not found.  Reading of raster files will be disabled.
For information about changes from previous versions of the mapping package, run 'news mapping'.

Here you notice the warning about GDAL libraries not being found, leading to reduced functionalities. So you can remove the package and install it again with the GDAL module loaded:

octave:4> pkg uninstall mapping
octave:5> exit
[dfr@lemaitre3 ~]$ ml GDAL
[dfr@lemaitre3 ~]$ octave
octave: X11 DISPLAY environment variable not set
[...]

octave:1> pkg install mapping-1.4.0.tar.gz
For information about changes from previous versions of the mapping package, run 'news mapping'.

The warning has disappeared and the package is available.

octave:2> pkg list
Package Name  | Version | Installation directory
--------------+---------+-----------------------
     control  |   3.1.0 | .../easybuild/2018b/software/Octave/4.4.1-foss-2018b/share/octave/packages/control-3.1.0
    geometry  |   4.0.0 | /home/ucl/pan/dfr/octave/geometry-4.0.0
          io  |  2.4.12 | .../arch/easybuild/2018b/software/Octave/4.4.1-foss-2018b/share/octave/packages/io-2.4.12
     mapping  |   1.4.0 | /home/ucl/pan/dfr/octave/mapping-1.4.0
     matgeom  |   1.2.3 | /home/ucl/pan/dfr/octave/matgeom-1.2.3
      signal  |   1.4.0 | .../easybuild/2018b/software/Octave/4.4.1-foss-2018b/share/octave/packages/signal-1.4.0
  statistics  |   1.4.0 | .../2018b/software/Octave/4.4.1-foss-2018b/share/octave/packages/statistics-1.4.0

Once the package is loaded, the azimuth function is found and behaves as in the examples of its documentation.

octave:3> pkg load mapping
octave:4> which azimuth
'azimuth' is a function from the file /home/ucl/pan/dfr/octave/mapping-1.4.0/azimuth.m
octave:5> azimuth([10,10], [10,40])
ans =  87.336
octave:6> azimuth([0,10], [0,40])
ans =  90
octave:7> azimuth(pi/4,0,pi/4,-pi/2,"radians")
ans =  5.3279
octave:8>

Julia

With Julia you can use the pkg package: [https://docs.julialang.org/en/v1/stdlib/Pkg/index.html]

julia> using Pkg
julia> Pkg.add("DataFrames")

Perl

For Perl can be used the local::lib module and the cpanm tool:

For cpanm to install modules locally, you need to setup the environment according to the output of the perl -Mlocal::lib command. You can set it interactively with

eval $(perl -Mlocal::lib)

and/or set it once and for all with

perl -Mlocal::lib >> ~/.bash_profile

Then you can simply run

cpanm Algorithm::Numerical::Shuffle

to install the Algorithm::Numerical::Shuffle module.

By default, this will install the modules in the ~/perl5 directory. Should you want to install them to another place, give that path as argument to the local::lib module. For instance:

perl -Mlocal::lib=mylibs/perl >> ~/.bash_profile

to install in the mylibs/perl directory.

Installing with Yum or Aptitude

Installing with Yum (RedHat, Fedora, etc.) or Aptitude (Ubuntu, Debian, etc.) or any other packager manager is not possible for users. Things like sudo apt-get install <name of package> will fail because all clusters run the CentOS distribution that does not use the Aptitude packager manager, and users are not allowed to use sudo (See below).

If your program can only be installed with apt-get, then you will need to use Singularity [https://www.sylabs.io/docs/].

Use of the sudo command

Do not try to use the sudo command; it will fail. Only local system administrators are able to gain root-level privileges. Regular users are not allowed to, simply because they would continuously break each other’s configuration, or potentially destroy the whole system. There is therefore no way root-level privileges will ever be granted to users.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Singularity Container

Singularity is a container solution for compute driven workloads. It is allowing to pack your application with all its dependencies such that you can run it out of the box in various different HPC clusters and computers.

More informations on Singularity:

Singularity web site: https://www.sylabs.io/singularity/

CECI formation on Singularity: https://indico.cism.ucl.ac.be/event/41/

Singularity is installed on the following CECI clusters:

lemaitre3

MPI

The portability of the application of MPI application is not trivial. The code compille within the container need to have the same slurm library and the same OpenMPI routine as the one installed on the cluster. For this reason we have prepared some Singularity image that you can use to create your singularity container.

Those containers can be found in /CECI/soft/src/singularity

Here is an example of an mpi hello-world application package in a container with the help of such container:

BootStrap: localimage
From: openmpi_2.1.1.simg

%runscript
    /usr/bin/mytest-mpi

%files
    test-mpi.c /opt/test-mpi.c

%post
    echo "Hello from inside the container"
    mpicc -o /usr/bin/mytest-mpi /opt/test-mpi.c

That you can then run on our clusters via:

srun -n 4 -p debug  bash -c "singularity run -B \$LOCALSCRATCH:/localscratch ./mpi-user.simg"

Note the “-B $LOCALSCRATCH/$SLURM_JOB_ID:/localscratch” which is needed to correctly run the mounting point in the singularity container. The use of the “bash -c” allows to postpone the evaluation of $LOCALSCRATCH up to a time where the environment variable is actually defined.

Note

The fakeroot feature of Singularity is not possible on the CECI cluster because it requires, several thousands UIDs to be allocated for every user, which is untractable given the number of CECI users.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Visualization server

CÉCI users have the possibility to launch a VNC session [https://en.wikipedia.org/wiki/Virtual_Network_Computing] in a dedicated server, with hardware acceleration, to run 3D visualisation applications.

Note

This service is only available in Hercules cluster

Start a VNC session

Connect to the cluster, execute the sviz command and answer the questions.

ceciuser@hercules:~$ sviz

Visualization node

    [1] - her2-i002 (default)

Please select a visualization node: 1

Waiting for a slot on a visualization node...

VNC server ready!

Connection information:

    VNC server: XXX.XXX.XXX.XXX:PPPP
    Password:   session_password
...

You will get the connexion information: the IP address XXX.XXX.XXX.XXX, the port number PPPP and the session password.

Connecting to the visualisation node

Connecting from Windows

First install MobaXterm as in indicated in the connexion setup page

Click on the Session icon session icon, then in the VNC icon vnc icon and select Network settings tab session icon

Set the IP address XXX.XXX.XXX.XXX and the port PPPP you get with sviz command. Tick the Connect through SSH gateway box and add the ssh gateway address of your university and in User add the login name to connect to the gateway.

UCL: gwceci.cism.ucl.ac.be with CÉCI login name

ULB: gwceci.ulb.ac.be with CÉCI login name

UMons: dragon2.umons.ac.be with CÉCI login name

UNamur: gwceci.unamur.be with UNamur eID

ULiège: gwceci.uliege.be with CÉCI login name

Tick the Use private key box and click on the icon session icon to browse and locate your id_rsa.ceci ssh key file.

Press the OK button and you should be prompted for your vnc session password.

You should now be connected to the vnc session

Connecting from Linux

Download TurboVNC [https://www.turbovnc.org/] from sourceforge [https://sourceforge.net/projects/turbovnc/files/] and install last version of the deb package or the rpm package (2.2.5 when writing this doc). If java is not installed in your computer, you will need to install it.

For example:

On Redhat/Centos like use:

# if needed install java
sudo yum install java-1.8.0-openjdk
# then install turbovnc
wget https://netcologne.dl.sourceforge.net/project/turbovnc/2.2.5/turbovnc-2.2.5.x86_64.rpm
sudo yum localinstall ./turbovnc-2.2.5.x86_64.rpm

On Debian/Ubuntu like use:

# if needed, install java
sudo apt-get install default-jre
# then install turbovnc
wget https://netix.dl.sourceforge.net/project/turbovnc/2.2.5/turbovnc_2.2.5_amd64.deb
sudo apt-get install ./turbovnc_2.2.5_amd64.deb

Connect to your VNC session with the command:

/opt/TurboVNC/bin/vncviewer XXX.XXX.XXX.XXX:PPPP -password 'session_password' -via gwusername@gwceci.server.be -SSHKeyFile /home/user/.ssh/id_rsa.ceci

Replace the IP address, the port and the session password you get with sviz command. Replace gwusername@gwceci.server.be by your login and university gateway

UCL: gwceci.cism.ucl.ac.be with CÉCI login name

ULB: gwceci.ulb.ac.be with CÉCI login name

UMons: dragon2.umons.ac.be with CÉCI login name

UNamur: gwceci.unamur.be with UNamur eID

ULiège: gwceci.uliege.be with CÉCI login name

and set the user the path to your ssh CÉCI key.

You will be asked for your ssh CÉCI key passphrase

Then you are connected to your VNC session, you can launch yours visualisation applications.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Jupyter

For heavy computations, you can run a Jupyterhub service in a Slurm job allocation on any cluster. Jupyterhub allows you, among others, to access a to a python notebook. To achieve this, you need to the following steps:

Find the module to load

Submit a jupyter job in a cluster

Connect your browser to the jupyterhub

Find the module to load

Connect to a cluster using ssh or MobaXterm.

Search for IPython or JupyterLab modules in the documentation or execute in a cluster one of this commands:

$ module spider IPython
$ module spider JupyterLab

Chose one of the available modules and use module spider again to get the module dependencies. Example:

module spider IPython/7.18.1-GCCcore-10.2.0
...
You will need to load all module(s) on any one of the lines below before the "IPython/7.18.1-GCCcore-10.2.0" module is available to load.

      releases/2020b
...

In this case you will need to load releases/2020b before loading IPython .

Submit a job to start jupyter

Once you know which module to load, submit a job to work for 30 min with the command:

$ srun -t 0:30:00 --pty bash -c 'ml releases/2020b; ml JupyterLab; jupyter notebook --ip $(hostname -i)'

The following have been reloaded with a version change:
  1) releases/2019b => releases/2020b

[I 11:09:07.956 NotebookApp] Serving notebooks from local directory: /home/users/-/-/ceciuser
[I 11:09:07.957 NotebookApp] Jupyter Notebook 6.1.4 is running at:
[I 11:09:07.957 NotebookApp] http://10.252.2.1:8888/?token=6392d08d83e5826e27f044a40973273e682282dcbc49894a
[I 11:09:07.957 NotebookApp]  or http://127.0.0.1:8888/?token=6392d08d83e5826e27f044a40973273e682282dcbc49894a
[I 11:09:07.957 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 11:09:07.990 NotebookApp] No web browser found: could not locate runnable browser.
[C 11:09:07.990 NotebookApp]

You need to keep the compute node IP address (10.252.2.1) the port (8888) and the connexion token for next step. Leave the terminal open until you have finish. To end the job use CTRL-C.

Note

For dragon2 and lemaitre3 you need to add the option --cluster=$CLUSTER_NAME. Example

srun -t 0:30:00 --cluster=$CLUSTER_NAME --pty bash -c 'ml load releases/2019b; ml load IPython/7.9.0-fosscuda-2019b-Python-3.7.4; jupyter notebook --ip $(hostname -i)'

Connect to the jupyterhub interface

Linux or MacOS

If you are using Linux or MacOS, then use the nice little tool named Sshuttle [https://sshuttle.readthedocs.io/en/stable/#] tool to access it.

Warning

sshuttle is currently not supported directly on Microsoft Windows [https://sshuttle.readthedocs.io/en/stable/windows.html].

Sshuttle is a program that, according to its documentation, acts as a “Transparent proxy server that works as a poor man’s VPN. Forwards over ssh. Doesn’t require admin. Works with Linux and MacOS.” Installation on a Linux or MacOS laptop can be done with apt, pacman, dnf depending on your Linux distribution, or with brew, MacPort on MacOS, but you can also install it with pip or git clone from GitHub directly. See the GitHub page [https://github.com/sshuttle/sshuttle] for details.

After you have installed sshuttle, make sure you can access the cluster using an SSH gateway or VPN if necessary. In the following, we will assume the connection to nic5 through one of the gwceci gateways is properly configured in .ssh/config as nic5.

Then, create a tunnel with sshuttle. Open a terminal window that will be dedicated to that and run the following in it.

$ sshuttle -r nic5 10.252.1.0/16
[local sudo] Password:
Warning: No xauth data; using fake authentication data for X11 forwarding.
client: Connected.

It will ask for your password in order to elevate privileges (sudo). That is needed because it will modify temporarily low-level network routing. As long as it runs, the private network of the cluster will be accessible from your laptop, wherever you are connected. Leave that terminal session open for as long as you need access to the Jupyter service in your job.

If your terminal supports it, you can then click on the URL that starts with http://10. in previous step. Otherwise simply copy that URL in your browser and you should see the Jupyterhub interface.

Once you are finished hit CTRL-C in both terminals to stop everything.

The IP range for sshuttle depends on the cluster.

nic5 10.252.1.0/16

lemaitre3 10.7.1.1/24

dragon2 10.102.169.0/24

See a video with a connexion example on lemaitre3. [https://medias.unamur.be/videos/use-jupyterhub-on-cluster-linux/]

Windows

If you are using Windows, install and configure MobaXterm.

Use the compute node IP address and port you get when submitting the job.

To reach the IP address of the computer node from your computer (localhost) two forwarding ports must be created. The first one will forward a intermediate localhost port (1111) to the front-end (nic5, lemaitre3 ...) through the gateway. The second one will forward the local port 2222 on the localhost port 1111 to the computer node port 8888

Create the first tunnel between your computer and the cluster frontend:

Open MobaXTerm and click on “tunnelling” tunnelling and click on “new ssh tunnel” new tunnel to create the first tunnel.

Fill the local port with 1111.

Fill “ssh server” with the login and gateway address of your university.

Fill the remote server with the frontend cluster address

Click Save and set the name of the new tunnel with the cluster name

Click on the key key and add your CÉCI key id_rsa.ceci

You can create a tunnel for each cluster (nic5, lemaitre3 ...)

Create the second tunnel between your computer and the compute node through the first tunnel:

Create a “new ssh tunnel” new tunnel.

Fill the local port with 2222.

Fill “ssh server” with the localhost forwarded port 1111 and your CÉCI user login name

Fill the remote server with the compute node IP and port you get when submitting the job.

Click Save and set the name of the new tunnel as node.

Click on the key key and add your CÉCI key id_rsa.ceci

You will have something like this:

Start the tunnel start tunnel for the cluster where you submitted the job and the tunnel for the node. Then open your browser at http://localhost:2222/?token=XXXXXXXXXX. Replacing XXXXXXXXXX with the token you received wen submitting the job.

See a video with a connexion example on dragon2. [https://medias.unamur.be/videos/use-jupyterhub-on-cluster-windows/]

Note

Each time you summit a new jupyterhub job, you could receive a different IP address and port. Replace the IP address and port in the node tunnel before start it.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Login shell

The login shell on all CÉCI clusters is Bash. This is defined in the central user directory and cannot be changed on the clusters with the usual chsh or chsh.ldap Linux commands.

There is however a workaround, illustrated here with zsh and tcsh. The first step is to check that the shell you want to use is installed on the cluster with the which command.

which zsh
which tcsh

When the shell is not installed, you can either request it to be installed by the system administrators or install it by yourself using the methods described earlier in this section.

Then you can add this to your .bash_profile file:

export SHELL=$(which zsh)
exec $SHELL -l

or

export SHELL=$(which tcsh)
exec $SHELL -l

This will result in the shell of your choice replacing the Bash process as soon as it starts. Exiting the shell will disconnect from the cluster as expected.

Warning

We do not offer support for the module system with another shell than Bash. If the module command does not work with the shell you chose, please refer to the lmod documentation [https://lmod.readthedocs.io/en/latest/030_installing.html#integrating-module-into-users-shells].

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Slurm F.A.Q

Q00 Can I bypass Slurm?

Q01 Is there a generic submission script I can use?

Q02 How can I get my job to start early?

Q03 How do I cancel a job?

Q04 How do I submit a job to a specific queue?

Q05 How do I create a parallel environment?

Q06 How do I choose a node with certain features (e.g. CPU, GPU, etc.) ?

Q07 How do I get the list of features and resources of each node ?

Q08 Is OpenMP ‘slurm-aware’ ?

Q09 Is MPI ‘slurm-aware’ ?

Q10 How do I know how much memory my job is using/has used ?

Q11 How do I use the local scratch space ?

Q12 How do I get the node list in full rather than in compressed format ?

Q13 How do I know which slots exactly are assigned to my job ?

Q14 How do I translate a script from another job scheduler into Slurm ?

Q15 When will my job start?

Q16 How do I know to which partition I should submit my job so that it starts as early as possible?

Q17 What are the queue settings and cpu resources available per user on each CÉCI cluster?

Q00 Can I bypass Slurm?

No. All computations must be submited as Slurm jobs. You are not allowed to run jobs on the login nodes (exception for little tests) ; you may also not connect to compute nodes by SSH outside of a Slurm allocation.

Warning

By using the CÉCI clusters, you accept the rules related to the Fair use of the CÉCI clusters [https://www.ceci-hpc.be/fair_use.html] .

Q01 Is there a generic submission script I can use?

See the Slurm Script Generation Wizard [https://www.ceci-hpc.be/scriptgen.html].

Q02 How can I get my job to start early?

First, make sure you only request the resources you need. The more you ask, the longer you will wait. Then, try and make your job flexible in terms of resources. If your program is able to work through the network, do not ask for all tasks on the same node. Use the Slurm options cleverly. For instance, the --nnodes option allows specifying a range of number of nodes, e.g. --nnodes=2-4, meaning that your job will start as soon as at least two nodes are available, but if, by then, four nodes are available, you will be allocated four nodes. Another example is –time-min, which allows you to specify a minimum running time which you are willing to set for your job if it makes it possible to start it earlier (through backfilling). Note that you can get, at any moment, the remaining time for your job in your script with the squeue -h -j $SLURM\_JOBID -o %L and that the --signal option can be used to be warned before the job is killed.

Of course, it is always easier to get a job running when there are few jobs waiting in the queue. Try to plan ahead your work and submit your jobs when people usually do not work on the cluster: holidays, exam periods, etc.

Q03 How do I cancel a job?

Use the scancel jobid command with the jobid of the job you want cancelled. In the case you want to cancel all your jobs, type scancel -u login. You can also cancel all your pending jobs for instance with scancel -t PD.

Q04 How do I submit a job to a specific queue?

Slurm uses the term partition rather than queue. To submit a job to a given partition, use the --partition option of the sbatch, srun, or salloc commands.

Queues can also appear under the form of qualities of service (QOS). To use a particular QOS, use the --qos option of the above-listed commands.

To view all partitions on a cluster, use sinfo, while qualities of service can be listed with sacctmgr list qos.

Q05 How do I create a parallel environment?

Slurm ignores the concept of parallel environment as such. Slurm simply requires that the number of nodes, or number of cores be specified. But you can have the control on how the cores are allocated; on a single node, on several nodes, etc. using the --cpus-per-task and --ntasks-per-node options for instance.

With those options, there are several ways to get the same allocation. For instance, the following : --nodes=4 --ntasks=4 --cpus-per-task=4 is equivalent in terms of resource allocation to --ntasks=16 --ntasks-per-node=4. But each one will lead to environment variables being set, and understood, differently by srun and mpirun: in the first case 4 processes are launched while in the second one 16 processes will be launched.

Suppose you need 16 cores, these are some possible scenarios:

you use mpi and do not care about where those cores are distributed: --ntasks=16

you want to launch 16 independent processes (no communication): --ntasks=16

you want those cores to spread across distinct nodes: --ntasks=16 --ntasks-per-node=1 or --ntasks=16 --nodes=16

you want those cores to spread across distinct nodes and no interference from other jobs: --ntasks=16 --nodes=16 --exclusive

you want 16 processes to spread across 8 nodes to have two processes per node: --ntasks=16 --ntasks-per-node=2

you want 16 processes to stay on the same node: --ntasks=16 --ntasks-per-node=16

you want one process that can use 16 cores for multithreading: --ntasks=1 --cpus-per-task=16

you want 4 processes that can use 4 cores each for multithreading: --ntasks=4 --cpus-per-task=4

Q06 How do I choose a node with certain features (e.g. CPU, GPU, etc.) ?

Slurm associates to each node a set of Features and a set of Generic resources. Features are immutable characteristics of the node (e.g. network connection type) while generic resources are consumable resources, meaning that as users reserve them, they become unavailable for the others (e.g. compute accelerators).

Features are requested with --constraint="feature1&feature2" or --constraint="feature1|feature2". The former requests both, while the latter, as one would expect, requests at least one of feature1 or feature2. More complex expressions can be constructed. Type man sbatch for details.

Generic resources are requested with --gres="resource:2" to request 2 resources.

Generic resources and features can be listed with the command scontrol show nodes.

Q07 How do I get the list of features and resources of each node ?

The command sinfo gives such information. You need to run it with specific output parameters though:

sinfo -o "%15N %10c %10m  %25f %10G"

It will output something like:

ceciuser@cecicluster:~ $ sinfo -o "%15N %10c %10m  %25f %10G"
NODELIST        CPUS       MEMORY      FEATURES                  GRES
mback[01-02]    8          31860+      Opteron,875,InfiniBand    (null)
mback[03-04]    4          31482+      Opteron,852,InfiniBand    (null)
mback05         8          64559       Opteron,2356              (null)
mback06         16         64052       Opteron,885               (null)
mback07         8          24150       Xeon,X5550                TeslaC1060
mback[08-19]    8          24151       Xeon,L5520,InfiniBand     (null)
mback[20-32,34] 8          16077       Xeon,L5420                (null)

In the above screen capture, we see that some compute nodes have Infiniband connections, some have Intel processors, while others have AMD processors. The node mback07 furthermore has one GPU, which is a generic resource in the sense that once requested by a job, it becomes unavailable for the others.

Q08 Is OpenMP ‘slurm-aware’ ?

No, you need to set export OMP_NUM_THREADS=... in your submission script.

For instance, if you requested several cores with the --cpus-per-task option, you can write:

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

Q09 Is MPI ‘slurm-aware’ ?

Yes, you do not need to specify the -np nor the -host, or hostfile options to mpirun or mpiexec. Simply go with

mpirun ./a.out

assuming you requested several cores with --ntasks

Do not forget to set the environment correctly with something like module load openmpi/gcc if necessary.

Q10 How do I know how much memory my job is using/has used ?

If your job is still running, you can have memory information with sstat. If your job is done, the information is provided by sacct. Both support the --format option so you can run, for instance:

sacct --format JobID,jobname,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize

See the manpages for both utilities:

man sstat
man sacct

Q11 How do I use the local scratch space ?

The CÉCI clusters are configured so that a temporary directory is created for each job on each node, and referenced to by the $LOCALSCRATCH environment variable. The temporary directory will typically be something like /scratch/<login_name>/<job_id>.

Therefore, in the case of a single-node job, you can simply cp or rsync the file or directory you need directly to $LOCALSCRATCH. For instance:

cp $HOME/inputfile $LOCALSCRATCH/
cd $LOCALSCRATCH
# do some work here

The $LOCALSCRATCH is automatically deleted at the end of the job. It is therefore necessary to copy back any result back to a global filesystem, either $HOME or $GLOBALSCRATCH for further processing or as a temporary storage before egressing to a long-term storage system.

cp -r $HOME/inputs $LOCALSCRATCH/
cd $LOCALSCRATCH
# do some work here
cp -r ./results $GLOBALSCRATCH

In the event of a multi-node job, Slurm offers the sbcast command that propagates a file to the local file systems of the nodes that were allocated to the job. However, sbcast works one file at a time.

One other way to deal with several files is to run something like srun cp .... Beware in that case that the srun command will run as many cp commands as the number of ntasks requested for the job, which could be larger than the number of allocated nodes, in which case multiple cp command will fight over access to the destination directory and might overwrite each other’s files. Make sure then to restrict the srun to the number of nodes that are allocated:

srun -n $SLURM_JOB_NUM_NODES cp -r $HOME/inputs $LOCALSCRATCH/

Q12 How do I get the node list in full rather than in compressed format ?

Slurm describes node lists with notations like node[05-07,09-17]. To get the full list, use the scontrol command:

ceciuser@cecicluster:~ $ scontrol show hostname node[05-07,09-17] | paste -d, -s
node05,node06,node07,node09,node10,node11,node12,node13,node14,node15,node16,node17

Q13 How do I know which slots exactly are assigned to my job ?

The command scontrol show -d job jobid gives very detailed information about jobs.

Q14 How do I translate a script from another job scheduler into Slurm ?

The Slurm developers maintain a ‘Rosetta stone of Workload manager [https://slurm.schedmd.com/rosetta.pdf]‘ which gives the correspondences between the options of several job schedulers. [Direct link - pdf 156kb [https://slurm.schedmd.com/rosetta.pdf]].

Q15 When will my job start ?

A job starts either when it has the highest priority and the required resources are available, or when it has an opportunity to backfill (See the document Slurm priorities for details). The squeue --start command gives an estimation of the date and time a job is supposed to start but beware that the estimation is based on the situation at a given time. Slurm cannot anticipate higher-priority jobs being submitted after yours, or machine downtimes which lead to fewer resources for the jobs, or job crashes which can lead to large jobs starting earlier than expected thus making smaller jobs scheduled for backfilling to lose that backfilling opportunity.

Q16 How do I know to which partition I should submit my job so that it starts as early as possible?

Simply submit the job to all the partitions you are considering, by listing them with the --partition option:

#SBATCH --partition=partition1,partition2

The job will be submitted to the partition which offers the earliest allocation according to your job parameters and priority.

Q17 What are the queue settings and cpu resources available per user on each CÉCI cluster?

The default settings are summarized on the cluster description page [https://www.ceci-hpc.be/clusters.html]. These settings are indicative only, and may change depending on the load on the clusters. To obtain current settings on a given queue, see Question 4.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Slurm priorities

Jobs are scheduled by Slurm according to their priority, this is not a first-come first-serve queue. We cannot stress this enough; the waiting time for a job is not related to the length of the queue; it is related to the user’s fairshare and to the size of the job. Even if the queue has 1000 jobs pending, your job could be starting right away if your fairshare is favorable, or if your job is so small it can be backfilled and scheduled in the shadow of a larger, higher-priority job. But of course if you do not submit your job, you have zero chance for your job to start... Priorities are computed per partition. Some clusters have special partitions with small max time that allow for fast job turnaround. On those partitions, your small jobs are even more akin to start soon. To view the partitions on the clusters, use the sinfo command, or consult the clusters details on the CÉCI website.

Slurm computes job priorities regularly and updates them to reflect continuous change in the situation. For instance, if the priority is configured to take into account the past usage of the cluster by the user, running jobs of one user do lower the priority of that users’ pending jobs.

The way the priority is updated depends on many configuration details. This document explains how to discover them and find the appropriate documentation so as to be able to understand how priorities are computed for a particular cluster.

Two parameters in Slurm’s configuration determine how priorities are computed. They are named SchedulerType and PriorityType.

Internal or external scheduling

The first parameter, SchedulerType, determines how jobs are scheduled based on available resources, requested resources, and job priorities. Scheduling can be taken care of by an external program such as Moab [http://en.wikipedia.org/wiki/Moab_Cluster_Suite] or Maui [http://en.wikipedia.org/wiki/Maui_Cluster_Scheduler], or by Slurm itself.

In that later case, the scheduling type can be builtin, in which case all jobs run in priority order, or backfill. Backfill is a mechanism by which lower priority jobs can start earlier to fill the idle slots provided they are finished before the next high priority jobs is expected to start based on resource availability.

To find out which solution is implemented on a cluster, you can issue the following command:

scontrol show config | grep SchedulerType

If the answer is sched/wiki, this means that scheduling is handled by Maui, while sched/wiki2 means scheduling is done by Moab. If scheduling is actually configured to be managed by Slurm, the above command should return sched/builtin or sched/backfill.

See the slurm.conf manpage [http://slurm.schedmd.com/slurm.conf.html] and search for ‘SchedulerType’ for more information.

If the scheduling is performed externally to Slurm (Maui or Moab), you will need to look for the proper documentation. If, as most likely, the scheduling is handled internally, the following section explains how to understand priority computations.

Priority computation

The way the priority is computed for a job depends on another parameter which is called PriorityType. It can take the following values:

priority/basic Jobs are given a strictly first arrived first served priority. (Mostly used in case of an external scheduler.)

priority/multifactor Jobs are prioritized according to several criteria such as past cluster usage, job size, queue time, etc.

priority/multifactor2 A variation of the previous.

To find out which solution is implemented on a cluster, you can issue the following command:

scontrol show config | grep PriorityType

The most used configuration is most probably priority/multifactor. The priority then depends on five elements:

Job age: how long the job has been waiting in the queue ;

User fairshare: a measure of past usage of the cluster by the user ;

Job size: the number of CPUs a job requests ;

Partition: the partition to which a job is submitted , specified with the --partition submission parameter;

QOS: a quality of service associated with the job, specified with the --qos submission parameter.

Note that the job age parameter is bounded so that priority stops increasing when the bound is attained. The job size parameter can be configured to favour small or large jobs, although it is used most of the time to favour large jobs. The fairshare parameter has a ‘forgetting’ parameter that leads to considering only the recent history of the user and not its total use over the time life of the cluster.

All these are combined in a weighted average to form the priority. The weights can be found by running

sprio -w

A detailed description of how these are computed (including the fairshare), is given in the Slurm documentation for multifactor [http://slurm.schedmd.com/priority_multifactor.html] and for sheduling [https://slurm.schedmd.com/SUG14/sched_tutorial.pdf].

The precise configuration for a cluster can be found by running the following command:

scontrol show config | grep ^Priority

Finding a user’s current fairshare situation is done with the sshare [http://slurm.schedmd.com/sshare.html] command.

Getting the priority given to a job can be done either with squeue [http://slurm.schedmd.com/squeue.html]

squeue -o %Q -j jobid

or with the sprio [http://slurm.schedmd.com/sprio.html] command which gives the details of the computation.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

An interactive job submission tool: ssubmit

Warning

The ssubmit tool is only available on Hercules2 cluster

Overview

For users who are not yet very experienced in using the SLURM batch system, we provide a tool, ssubmit, that simplifies job submission. It generates the sbatch script and submits the job.

Basic Usage

A batch job can be submitted using the ssubmit command

ssubmit program arg1 arg2 ...

where:

program is the application you want to run

arg1 arg2 ... are the program arguments (if any)

If your programs is from a global installed application, you need to load the module before use ssubmit.

module load application
ssubmit program arg1 arg2 ...

Then, ssubmit will ask you for the resources reserved for your job in terms of time, memory and number of cores. ssubmit creates a submission script and submits your job. The submission script is stored in a slurm-JOB_ID.sh file and the job output in a slurm-JOB_ID.out file, where JOB_ID is the job ID set by slurm

The following example will submit a job that executes the echo command with 'hello from ...' as argument on a compute node using 1 core with 1 Gb of memory during 1 hour.

ssubmit echo ‘hello from job id $SLURM_JOB_ID running on compute node $(hostname)’

====== Time ====================================================================

  [1]  1 hour (default)
  [2]  1 day
  [3]  5 days
  [4]  15 days (max)

  Your choice or your time (DD-HH:MM or HH:MM): 1

  Using serial paradigm 1 process

====== Memory ==================================================================

  Memory per process in GB (defaul 1GB  max 2000GB): 1

====== Summary =================================================================

  You are about to submit a job to the cluster.
  Please check if everything is correct.

                Job name: echo
              Executable: echo
               Arguments: ‘hello from job id running on compute node cecicluster’
    Number of processors: 1
    Memory per processor: 1024
                    Time: 00-01:00:00
               Partition: batch

  Would you like to continue [Y/n] ?y

====== Submitted job informations ==============================================

                  Job id: 201403967
              Job output: slurm-201403967.out
     Get job status with: squeue -j 201403967

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
         201403967     batch     echo ceciuser PD       0:00      1 (None)

In this case, the job output is located in slurm-201403967.out and the submission script is stored in slurm-201403967.sh

Ssubmit arguments

You can set up some job parameters using ssubmit’s arguments. The arguments must been set before program name. Those are:

-y: Automatic confirm of submission values

-t  --time: TIME: allocation time DD-HH:MM

-m  --mem-per-cpu: MEMORY: Memory per process in MB

-n  --ntasks: NTASKS: Number of process

-J  --job-name: NAME: Specify a name for the job allocation

-p , --template-path: PATH: Specify a template files path

For example:

ssubmit --time DD-HH:MM --mem-per-cpu MB --ntasks N --job-name NAME program arg1 arg2 ...

The -y option allow you to skip the confirmation step

ssubmit --time 00-01:00 --mem-per-cpu 1024 -y echo 'hello world'

Advanced usage

You can modify the way ssubmit generates the submission script adding your own template files in a folder determined with the -p argument.

In the templates directory, you can add three kind of files that will overwrite the default ones.

template.js

This is a jinja2 [https://jinja2docs.readthedocs.io/en/stable/] file with the content that ssubmit will write after the #SBATCH header in the submission script. In the jinja2 [https://jinja2docs.readthedocs.io/en/stable/] template you can access this variables:

path: the directory path where your application is located

cmd: the program name

argv: an array with the command and the arguments.

sbatch: a dict with the sbatch parameters

inc: a dict with user defined variables (see inc.py module)

paradigm: the selected paradigm.

config: a dict with the data you set in config.yaml file

The default template is

{# minimal template #}
{{ path }}/{{ cmd }} {% for i in argv[2:] %}{{ i }} {% endfor %}

An example using inc and sbatch variables:

echo "inc variables example p1={{ inc.p1 }}, p2={{ inc.p2 }}"
hostname

mpirun -np {{ sbatch['ntasks'] }} {{ cmd }}

config.yaml

By default, the job paradigm is set to serial so your job will only use one core.

For preinstalled applications supported paradigm are predefined.

If you want to overwrite the predefined paradigms or set the paradigms for applications you have installed in you home directory, you need a yaml [https://en.wikipedia.org/wiki/YAML] file config.yaml that contains the paradigms your application is able to use.

For example, the config.yaml file for an application that support all allowed paradigms is

---

# List of paradigm this program supports
paradigms:
  - serial
  - smp
  - mpi

inc.py

inc.py is a python file or module that allows you to implement your own arguments checks and add new variables to the generated submission script. The functions you can define in this file are:

function
description

def check_args(argv)

Add the checks and modifications you need to your program arguments.

input: array with the arguments

output: array with the checked and/or modified arguments

def inc(argv)

Include new variables needed on your application.

input: array with the arguments

output: dict or list inc with new variables

check_sbatch(sbatch)

Used to check the sbatch parameters. Must return True or False depending if there is an error or not in the parameters

input: sbatch dict

output: boolean True or False

def get_sbatch_options(argv):

Append new sbatch parameters.

input: array with the arguments

output: a dict with {‘sbatch_variable’: ‘Value’, ...}

An example of inc.py could be:

import os
import sys

def check_args(argv):
    """ check specific arguments related to each application
        and do modifications on it if needed
        # argv[0] = ssubmit
        # argv[1] = application executable
        # argv[2...] = application arguments
    """

    # argument must be a input file
    if not os.path.isfile(argv[2]):
        print("ERROR: Input file %s doesn't exist or is not readable" % (argv[2]))
        sys.exit(1)

    return argv

def inc(argv):
    """ do any action related to the application and return variables as dict
        or list to be used in the sbatch template
    """

    return {'app_version': "20210630-R1"}

def check_sbatch(sbatch):
    """ Check sbatch """

    if sbatch['paradigm'] == 'mpi':
        if int(sbatch['ntasks']) % 2 == 0:
            return True
        else:
            print('Number of mpi process must be EVEN number')
        return False

def get_sbatch_options(argv):
    """ set more sbatch options if application needs it return a dict with
        { option: value, ... }
    """

    # Set log file name as input file name
    sbatch = {'output': argv[2].rsplit('.', 1)[0] + '.log'}

    return sbatch

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Getting Job Efficiency

To optimize the clusters usage, it is necessary that your reservation fits the memory, the CPU and the expected time to the needs of your job.

You can check the usage of memory and CPU for either running or completed jobs to calculate their efficiency.

Running jobs

For running jobs, use squeue to get the compute node name where your job is running.

$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
         202232023     batch    numpy ceciuser  R       0:36      1 'NODEID'

In this case, it is NODEID. Then use the top command through an ssh connexion to the compute node to get the CPU usage and the process ID:

$ ssh -t NODEID top -u $USER -b -n 1
Warning: Permanently added NODEID,192.168.1.107 (ECDSA) to the list of known hosts.
top - 17:23:27 up 262 days,  2:01,  0 users,  load average: 29.59, 30.05, 32.05
Tasks: 478 total,  26 running, 238 sleeping,   0 stopped,   0 zombie
%Cpu(s): 57.4 us,  0.6 sy,  0.0 ni, 41.0 id,  1.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 26379747+total, 21854316+free, 16868692 used, 28385624 buff/cache
KiB Swap:  4194300 total,  3826884 free,   367416 used. 24502859+avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
'11154' ceciuser  20   0 1504856 1.027g  14964 R '752.9'  0.4   6:40.75 python
 11233  ceciuser  20   0  172696   5092   4164 R    5.9   0.0   0:00.02 top
 11130  ceciuser  20   0  113856   3668   2888 S    0.0   0.0   0:00.01 slurm_scr+
 11232  ceciuser  20   0  168620   4596   3280 S    0.0   0.0   0:00.00 sshd

In this example the CPU usage is 752.9% and the process ID (PID) is 11154. Use the PID to get the memory usage with this command

$ ssh NODEID cat /proc/11154/status | grep VmRSS
Warning: Permanently added NODEID,192.168.1.107 (ECDSA) to the list of known hosts.
VmRSS:  '1076752 kB'

The CPU efficiency is 752.9÷800×100 = 94.1% and the memory efficiency is (1076752÷1024)÷1056×100 = 99.6%

In this example, the reservation was 8 CPU and 1056 MB

#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=132m

Completed jobs

For completed jobs you can use the seff command to get the efficiency, using the job ID as argument.

$ seff 202232023
Job ID: 202232023
Cluster: clusername
User/Group: ceciuser/ceciuser
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 8
CPU Utilized: 00:16:25
CPU Efficiency: '76.95%' of 00:21:20 core-walltime
Job Wall-clock time: 00:02:40
Memory Utilized: 1.03 GB
Memory Efficiency: '99.91%' of 1.03 GB

In the job example, after completion, we get 76.95% CPU efficiency and 99.91% memory efficiency.

Final remarks

You can also get the efficiency of a running or completed job with the sacct command.

$ sacct -j 202232023 -o "User,JobID%20,ReqMem,ReqCPUS,TotalCPU,Elapsed,MaxRSS,State"
     User                JobID     ReqMem  ReqCPUS   TotalCPU    Elapsed     MaxRSS      State
--------- -------------------- ---------- -------- ---------- ---------- ---------- ----------
 ceciuser            202232023      132Mc        8  16:25.207   00:02:40             COMPLETED
               202232023.batch      132Mc        8  16:25.207   00:02:40   1080420K  COMPLETED

MaxRSS/(ReqMem*ReqCPUS) gives the maximum of memory efficiency. In this example, (1080420)÷(132×8×1024)*100 = 99.91%

TotalCPU/(Elapsed*ReqCPUS) gives the CPU efficiency. In this example, (16×60+25,207)÷((2×60+40)×8)×100 = 76.95%

Note

Slurm checks periodically for the memory usage to get the “Maximum resident set size” of all tasks in job. If your code has a short peak usage of memory slurm will not see it so the value will be underestimated.

If your memory efficiency is bad, you should set the requested memory a little larger than the MaxRSS. Also, see if it is possible to estimate the memory usage based on a parameter, such as the grid size, matrix size, size of the big bunch of data read from a file ...

Several reasons can be the cause of a bad CPU efficiency

Some jobs have a pre-compute or post-compute part that uses less CPU. You can see if it is possible to split the calculations into several dependent jobs.

Some jobs creates multiple threads but only some of them are in “Running” status and the others are in “Sleep” status. This may be checked periodically while the job is running with the ssh -t NODEID top -H -n 1 -p PID command, where PID is the process ID (see above).

$ ssh -t NODEID top -H -n 1 -p 11154
Warning: Permanently added NODEID,192.168.1.107 (ECDSA) to the list of known hosts.
top - 09:36:43 up 62 days, 22:10,  1 user,  load average: 23.14, 23.09, 23.08
Threads:   4 total,   3 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s): 64.8 us,  0.8 sy,  0.0 ni, 28.2 id,  6.2 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 26378952+total, 18337344+free, 46629468 used, 33786600 buff/cache
KiB Swap:  4194300 total,  4111604 free,    82696 used. 21539291+avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
113551 ceciuser  20   0 4826492   3.1g   2364 R 99.9  1.2 653:34.06 python
 11154 ceciuser  20   0 4826492   3.1g   2364 D 99.9  1.2   1106:26 python
113552 ceciuser  20   0 4826492   3.1g   2364 R 99.9  1.2 653:18.76 python
113553 ceciuser  20   0 4826492   3.1g   2364 R 99.9  1.2 652:35.98 python

In this example, 3 threads are running and 1 is sleeping, as indicated by the third line of the output.

You can reduce the number of reserved CPU to the number of running threads.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Slurm limits

The main resource for understanding limits in a Slurm context is the “Resource limits” page from the documentation [https://slurm.schedmd.com/resource_limits.html].

Limits can be set at mutliple levels. They can be global, applying to all the cluster and all the users, or it can be specific to a partition or an account. Also, it can be specific to a quality of service (QOS). The following sections explains them in detail.

Contents

Slurm limits

Global limits

QOS limits

Account limits

Partition limits

Precedence

Global limits

Limits can be imposed globally, at the cluster level, for all users, ...

dfr@nic5-login1 ~ $ sacctmgr show cluster format=name,GrpJobs,GrpSubmit,GrpTRES,GrpTRESMins,GrpTRESRunMins,GrpWall,MaxJobs,MaxSubmit,MaxTRESMins,MaxTRES,MaxWall
       Name GrpJobs GrpSubmit       GrpTRES   GrpTRESMins GrpTRESRunMin     GrpWall MaxJobs MaxSubmit   MaxTRESMins       MaxTRES     MaxWall
 ---------- ------- --------- ------------- ------------- ------------- ----------- ------- --------- ------------- ------------- -----------

GrpJobs: Maximum number of running jobs in aggregate for this cluster

GrpSubmit: Maximum number of jobs which can be in a pending or running state at any time in aggregate

GrpTRES: Maximum number of TRES (“trackable resource [https://slurm.schedmd.com/tres.html]”) running jobs are able to be allocated in aggregate

GrpTRESMins: The total number of TRES minutes that can possibly be used by past, present and future jobs running on the cluster

GrpTRESRunMin: Used to limit the combined total number of TRES minutes used by all jobs running on the cluster

GrpWall: Maximum wall clock time running jobs are able to be allocated in aggregate for this cluster

MaxJobs: Maximum number of jobs each user is allowed to run at one time

MaxSubmit: Maximum number of jobs the cluster can have in a pending or running state at any time

MaxTRESMins: Maximum number of TRES minutes each job is able to use in this cluster

MaxTRES: Maximum number of TRES each job is able to use

MaxWall: Maximum wall clock time each job is able to use

... or user per user.

dfr@nic5-login1 ~ $ sacctmgr list user dfr withassoc format=user,maxjobs,maxnodes,maxTRES,maxSubmit,maxwall,maxTRESmins
      User MaxJobs MaxNodes  MaxTRES MaxSubmit     MaxWall  MaxTRESmins
---------- ------- -------- -------- --------- ----------- ------------
       dfr

MaxJobs: Maximum number of jobs this user is allowed to run at one time

MaxNodes: Maximum number of nodes per job

MaxTRES: Maximum number of TRES per job

MaxSubmit: Maximum number of jobs the user can have in a pending or running state at any time

MaxWall: Maximum wall clock time each job of this user is able to use

MaxCPUMins: Maximum number of CPU.minutes each job of this user is able to use

QOS limits

Quality of Services are the most versatile way of setting specific privileges, but also limits, to users and jobs. The following command lists them, along with their associated limits.

 dfr@nic5-login1 ~ $ sacctmgr list qos  format=name,GrpTRES,GrpTRESMins,GrpTRESRunMin,GrpJobs,GrpSubmit,GrpWall,MaxTRES,MaxTRESPerNode,MaxTRESMins,MaxWall,MaxTRESPU,MaxJobsPU,MaxSubmitPU,MaxTRESPA,MaxJobsPA,MaxSubmitPA,MinTRES
      Name       GrpTRES   GrpTRESMins GrpTRESRunMin GrpJobs GrpSubmit     GrpWall       MaxTRES MaxTRESPerNode   MaxTRESMins     MaxWall     MaxTRESPU         MaxJobsPU MaxSubmitPU     MaxTRESPA MaxJobsPA MaxSubmitPA       MinTRES
---------- ------------- ------------- ------------- ------- --------- ----------- ------------- -------------- ------------- ----------- ------------- --------- ----------- ------------- --------- ----------- -------------
    normal                                                                               cpu=320                                                cpu=384         384         640

GrpTRES: Maximum number of TRES running jobs are able to be allocated in aggregate for this QOS

GrpTRESMins: The total number of TRES minutes that can possibly be used by past, present and future jobs running from this QOS

GrpTRESRunMin: The total number of TRES minutes that can be used by all jobs running with this QOS

GrpJobs: Maximum number of running jobs in aggregate for this QOS

GrpSubmit: Maximum number of jobs which can be in a pending or running state at any time in aggregate with this QOS

GrpWall: Maximum wall clock time running jobs are able to be allocated in aggregate for this QOS

MaxTRES: Maximum number of TRES minutes each job is able to use in this QOS

MaxTRESPerNode: Maximum number of TRES each node in a job allocation can use.

MaxTRESMins: Maximum number of TRES minutes each job is able to use in this QOS

MaxWall: Maximum wall clock time each job is able to use in this QOS

MaxTRESPU: Maximum number of TRES each user is able to use

MaxJobsPU: Maximum number of jobs each user is allowed to run at one time

MaxSubmitPU: Maximum number of jobs pending or running state at any time per user.

MaxTRESPA: Maximum number of TRES each account is able to use

MaxJobsPA: Maximum number of jobs each account is allowed to run at one time.

MaxSubmitPA: A Maximum number of jobs pending or running state at any time per account

MinTRES: Minimum number of TRES each job running under this QOS must request. Otherwise the job will pend until modified.

In the example, a limit is set at 384 CPUs maximum per user in total, along with a 320 CPUs limit for each job, a maximum of 384 running jobs per user, and 640 maximum running or pending jobs per user.

Users can specifiy a chosen QOS for each job, otherwise the default cluster QOS (normal) applies unless a default user QOS...

dfr@nic5-login1 ~ $ sacctmgr list user dfr format=name,defaultqos
      Name   Def QOS
---------- ---------

... or default account QOS is set.

dfr@nic5-login1 ~ $ sacctmgr list account ceci format=name,defaultqos
         Name   Def QOS
   ---------- ---------

The above two example show no default QOS set at the user or account level.

Account limits

Accounts can have limits set too.

dfr@nic5-login1 ~ $ sacctmgr list account ceci withassoc where user=dfr format=account,GrpJobs,GrpNodes,GrpTRES,GrpMem,GrpSubmit,GrpWall,GrpTRESMins,MaxJobs,MaxNodes,MaxTRES,MaxSubmit,MaxWall,MaxTRESMin
    Account GrpJobs GrpNodes       GrpTRES  GrpMem GrpSubmit     GrpWall   GrpTRESMins MaxJobs MaxNodes       MaxTRES MaxSubmit     MaxWall   MaxTRESMins
 ---------- ------- -------- ------------- ------- --------- ----------- ------------- ------- -------- ------------- --------- ----------- -------------
       ceci

GrpJobs: Maximum number of running jobs in aggregate for this account and all accounts which are children of this account

GrpNodes: Maximum number of nodes running jobs are able to be allocated in aggregate for this account and all accounts which are children of this account.

GrpTRES: Maximum number of TRES running jobs are able to be allocated in aggregate for this account and all accounts which are children of this account.

GrpMem: Maximum amount of memory running jobs are able to be allocated in aggregate for this account and all accounts which are children of this account.

GrpSubmit: Maximum number of jobs which can be in a pending or running state at any time in aggregate for this account and all accounts which are children of this account.

GrpWall: Maximum wall clock time running jobs are able to be allocated in aggregate for this account and all accounts which are children of this account.

GrpTRESMins: Maximum number of TRES minutes running jobs are able to be allocated in aggregate for this account and all accounts which are children of this account.

MaxJobs: Maximum number of jobs this account is allowed to run at one time. This is overridden if set directly on a user. Default is the cluster’s limit.

MaxNodes: Maximum number of nodes per job the children of this account can run

MaxTRES: Maximum number of TRES each job is able to use in this account

MaxSubmit: Maximum number of jobs which this account can have in a pending or running state at any time.

MaxWall: Maximum wall clock time each job is able to use in this account

MaxTRESMin: Maximum number of TRES minutes each job is able to use in this account

Every user has a default account, but can also possibly choose from a set of accounts they have access to, for each job.

Partition limits

Each partition in a cluster has a set of limits specific to partitions, for instance here with a 2 day maximum wall time on the first two partitions and a quite larger walltime on the last one ...

dfr@nic5-login1 ~ $ scontrol show partitions  | egrep -ie "^P|Max"
PartitionName=batch
   MaxNodes=UNLIMITED MaxTime=2-00:00:00 MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
PartitionName=hmem
   MaxNodes=UNLIMITED MaxTime=2-00:00:00 MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
PartitionName=bio
   MaxNodes=UNLIMITED MaxTime=62-00:00:00 MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED

MaxTime: Maximum run time limit for jobs.

MaxCPUsPerNode: Maximum number of CPUs on any node available to all jobs from this partition.

MaxMemPerCPU: Maximum real memory size available per allocated CPU

MaxMemPerNode: Maximum real memory size available per allocated node

... and can be associated with a QOS, from which it will inherit all limits:

dfr@nic5-login1 ~ $ scontrol show partitions  | egrep -ie "^P|Qos"
PartitionName=batch
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=YES QoS=N/A
PartitionName=hmem
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=NO QoS=N/A
PartitionName=bio
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=NO QoS=N/A
dfr@nic5-login1 ~ $

In the above example, QoS=N/A shows there is no PartitionQOS set on any partition.

A summary for all limits related to a user and their account and the QOSes can be obtained with scontrol show assoc_mgr.

Precedence

The order in which limits are enforced is

Partition QOS limit

Job QOS limit

User association

Account association(s), ascending the hierarchy

Root/Cluster association

Partition limit

by default. If the configuration flag OverPartQOS is set, the order for job QOS and partition QOS is reversed.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Workflow management

Whenever you have to deal with multiple jobs on a HPC system, the idea of automating parts or all of the job management process involves describing and implementing so-called ‘workflows’. Options for managing workflows are numerous and range from using basic scheduler features such as job arrays and job dependencies, up to using a complex system backed up by a central, multi-user, database.

Important

Most topics presented here are addressed in the videos of the 2021 PRACE/EuroCC Workshop on Workflows available on the CISM/CECI Youtube channel [https://www.youtube.com/playlist?list=PLL1UN3Ywr5380HTONpdMs09nohnQa6Zzm].

Workflow management systems can typically do some (or all) of the following:

Compute dependencies between tasks and organise work ;

Submit jobs to the scheduler ;

Generate job scripts (templating, sweeping, etc.) ;

Install necessary scientific software ;

Monitor jobs and recover from failures, fault detection, “smart” reruns ;

Handle data : streaming, staging, etc. ;

Log processes and track data provenance ;

Enable sharing of data, results, workflows, with security and monitoring of access policies ;

Provide performance analysis and prediction.

Note

Workflow management systems have been developed in many scientific and technical communities. We will focus on the ones that are useful in a cluster computing context.

The list of existing workflow management systems is very long [https://github.com/common-workflow-language/common-workflow-language/wiki/Existing-Workflow-systems] (300+ items!). On top of that, some potentially less-known UNIX commands can also be very useful to manage jobs. This document aims at offering users with some guidance on the choice of the tool to use.

We will focus on tools that are

relevant to the HPC environments (not cloud, K8s, Hadoop, etc.) ;

standalone (not language-specific libraries) ;

using a simple domain-specific language (no XML or other convoluted language) ;

general purpose (not reserved to ‘omics’ for instance) ;

mature, easy to install, and have an active community.

Contents

Workflow management

Introduction

Wide workflows

Slurm tips

Helper tool: atools

Deep workflows

Slurm tips

sdag

Auto-requeuing job

Slurm tips

Maestro

Generic DAGs

SlurmDagman

Makeflow

Generic Cyclic workflows

Full-featured systems

Further reading

Introduction

Workflows exist in many areas of business, healthcare, administration, IT operations, machine learning, Internet of Things, etc. where specific sets of tools are available, some of which can be useful in a HPC context, but will not be addressed here.

From a cluster computing point of view, a workflow will be considered to be a collection of possibly interdependent jobs that are all part of the same study.

If we represent jobs with a dot, and dependency between jobs with an arrow, and arrange them from top to bottom, we can distinguish between “wide”, “deep”, and “cyclic” workflows.

Wide workflows are composed of many jobs that are actually independent one from another, and a few dependent jobs, for instance for consolidating the results. Deep workflows are composed of many jobs that are dependent on only one previous job. Cyclic workflows are more specific in the sense that they express the dependency of a job onto itself, which must be understood as the same job being re-submitted multiple times. For each type of workflow, specific techniques and tools exist.

Wide workflows

Typical studies include:

parameter sweep: each job runs an experiment with a specific value of a parameter and the aim is to find the optimum of a target variable ;

file collection processing ; each job processes a data file independently either to transform it or extract a summary value from its data ;

line-in-file processing ; each job runs an experiment using parameters values stored in a file listing all experiments.

Most wide workflows are made of multiple independent jobs and one “merging” or “summarizing” job, that often is performed interactively rather than submitted to the scheduler.

Slurm tips

The canonical way to deal with a wide workflow in Slurm is through job arrays [https://slurm.schedmd.com/job_array.html]. Job arrays are very useful to manage multiple jobs at once, but they are meant for jobs that - have a non negative bounded integer parameter, - but are otherwise completely identical.

The main option to deal with a parameter that does not fit the format of the parameter expected by Slurm (e.g. a list of files, a sequence of real values, etc.) is to use a Bash array. The template submission script would look like this:

#SBATCH --array=0-8 # Creates a 9-jobs array, ranked 0 to 8
FILES=(*.csv)       # Creates a Bash array with the list of CSV files in the working directory
PARAMS=(0.1 0.2 0.3 0.1 0.2 0.3  0.1 0.2 0.3)
                    # Creates a Bash array with real-valued numbers
./program --param=${PARAMS[$SLURM_ARRAY_TASK_ID]} ${FILES[$SLURM_ARRAY_TASK_ID]}
                    # Runs program on file whose order corresponds to the job rank in the array
                    # with a parameter whose value corresponds to the nth element in the PARAMS
                    # array, with n being the rank of the job in the array.

When the jobs are not identical, for instance the number of CPUs increases with the parameter as would be natural in a scaling study, the submission can be automated with Linux tools such as xargs or GNU Parallel taking advantage of the --wrap option of sbatch. Examples:

$ find . -name \*.csv | xargs -I {} sbatch --wrap "./program {}"
                  # Submit a series of jobs that run the program on each CSV file found in the current directory
$ parallel sbatch --wrap "./program --param={}" ::: 0.1 0.2 0.3 0.1 0.2 0.3  0.1 0.2 0.3
                  # Submit a series of jobs that run the program for each value in the list

More information: WorkflowsWithBasicTools.pdf [https://indico.cism.ucl.ac.be/event/119/contributions/46/attachments/124/278/WorkflowsWithBasicTools.pdf]

Helper tool: atools

To further alleviate the shortcomings of native Slurm job arrays, the tool suite atools comes very handy. The atools are a collection of utilities that allow you to better manage the parameters of the jobs, and also summarize the outcome of the jobs, identify and re-submit failed jobs, etc.

Tools include

aenv sets up job parameters as environment variables based on the job rank and a file describing all parameter values

alog fills a log file tracking which jobs in the array completed, failed, or are still pending, etc.

arange reports on the job array progress and can also re-submit failed jobs

areduce merges outputs of all jobs in the array into a “reduced” values, a la Hadoop

More information: https://atools.readthedocs.io/en/latest/

Deep workflows

Typical use cases:

pre- and post- processing jobs: jobs are necessary besides the main job to prepare the inputs and wrap up the outputs.

data transformation pipelines: each job runs a step in a transformation pipeline consuming data from the previous step and making it digestible for the next one.

Such jobs are dependent on one another ; job N+1 cannot start before job N has successfully completed.

Slurm tips

The canonical way to instruct Slurm about dependencies between jobs is with the --dependency option as in the following examples:

$ sbatch preprocess.sh
submitted batch job 1
$ sbatch -d afterok:1 process1.sh
submitted batch job 2
$ sbatch -d afterok:1 process2.sh
submitted batch job 3
$ sbatch -d afterok:2:3 postprocess.sh
$ sbatch -d afterany:1:2:3 cleanup.sh
$ sbatch -d afternotok:1?afternotok:2?afternotok:3 cancel.sh

One problem is that the dependee job ID must be known before submitting the dependent job (the job names unfortunately cannot be used). In the case of a linear dependency, it can be automated with a script like the following (chainsubmit.sh):

#!/bin/bash

 ID=$(sbatch --parsable $1)
 shift
 for script in "$@"; do
   ID=$(sbatch --parsable --dependency=afterok:${ID%%;*} $script)
 done

That script can then be used to submit multiple chained job scripts:

./chainsubmit.sh job1.slurm job2.slurm job3.slurm job4.slurm

Here is a variation of that script to use when the jobs submission script job.slurm accepts a parameter:

#!/bin/bash

SCRIPT=${1?Usage: $0 script.slum arg1 arg2 ...}
shift
ARG=$1
ID=$(sbatch --parsable $SCRIPT $ARG)
shift
for ARG in "$@"; do
  ID=$(sbatch --job-name=$ARG --parsable --dependency=afterok:${ID%%;*} $SCRIPT $ARG)
done

Use it like this:

./chainsubmit.sh job.slurm 2001 2002 2003 2004

to submit the job.slurm with four arguments, from 2001 to 2004.

If the steps in the workflow are short enough to hold within the maximum allowed walltime in the cluster, you can also pack them inside a single submission script, which will effectively run them in order. But in that case, you can also use GNU Make [https://www.gnu.org/software/make/] (the same make utility you probably already have used in the past to build software, its primary purpose) to build complex workflows with dependencies. Just create a Makefile and invoke the make command inside the submission script. Make sure to use the -j option of Make to allow running independent steps in parallel and make it compatible with the number of CPUs you requested.

More information: WorkflowsWithBasicTools.pdf [https://indico.cism.ucl.ac.be/event/119/contributions/46/attachments/124/278/WorkflowsWithBasicTools.pdf]

sdag

To alleviate the shortcomings of native Slurm job arrays, the sdag tool comes very handy. It uses a workflow description syntax similar to DagMan, a well-known tool in HTC, related to HTCCondor.

# File flow.dag
JOB A1 jobA.sbatch
JOB B2 jobB.sbatch
JOB C3 jobC.sbatch
JOB D4 jobD.sbatch

PARENT A1 CHILD B2 C3
PARENT B2 C3 CHILD D4

The above example expresses the dependencies between four jobs (A1, B2`, C3, D4), described by four distinct submission scripts (jobA.sbatch, ..., jobD.sbatch) as a diamond-shaped graph:

By running sdag flow.dag, you will submit all four jobs with the appropriate --dependency=afterok... options.

Some caveats though: sdag is does not handle federated clusters, does not handle re-submissions, and requires Python 2.7.

More information: https://github.com/abdulrahmanazab/sdag

Auto-requeuing job

Typical use case:

studies that are too long for the maximum wall time of the clusters (see the checkpointing video tutorial [https://www.youtube.com/watch?v=LyWSYYj4zHY&list=PLL1UN3Ywr53_FJWYozVaCbmSJgynibWQ_&index=18])

studies that cannot be parallelized efficiently

Slurm tips

There are two main problems to resolve in this case:

make sure the program stops in time for the wrap up operations (checkpoint) to perform properly (this includes possibly letting the program run until the next iteration) , and

make the job re-enter the queue.

The two main options for stopping the program are

using the timeout command that runs a command for a maximum duration. Use this when the program can regularly write its status on disk and can be interrupted without notification. Set a duration a bit shorter than the requested run time.

using UNIX signals [https://en.wikipedia.org/wiki/Signal_(IPC)]. Use this when the program is designed to handle signals and perform a status dump on disk upon receiving one. Slurm will send a signal if you specify the --signal option.

When the program has stopped properly, and is ready to run again (restart), the job can be submitted to the queue again. You can either

re-submit a new, identical, job with sbatch $0

re-queue the same job (and keep the same job ID) with scontrol requeue $SLURM_JOB_ID. Restarted jobs will have the environment variable SLURM_RESTART_COUNT set to the number of times the job has been restarted.

In both cases, you might want to use the --open-mode=append option to prevent Slurm from truncating the output file upon job restart.

Note that Slurm can be configured to automatically re-queue a job if it terminates with a specific return code ; see the RequeueExit and RequeueExitHold configuration options.

Maestro

Maestro removes the need to write the submission script all together and rather expect a Yaml description of the set of jobs that must be run, along with inline documentation of the study. Maestro then submits the jobs and is able to report on the progress, and organises the outputs nicely so that files from distinct studies do not mix together.

An example of study is as follows: ~ .. code:

description:
    name: maestro_test
    description: A simple study to run a program on files

study:
    - name: run_program
      description: run program on the files
      run:
         cmd: |
            ./program --param=$(PARAM)

global.parameters:
    PARAM:
        values: [0.1 0.2 0.3 0.1 0.2 0.3 0.1 0.2 0.3]
        label: PARAM.%%

More complete example can include multiple tasks with inter-dependencies, and Slurm-specific options for some of the tasks. Maestro also provides a utility framework pgen and pargs to be build arbitrary parameter collections using the Python language.

A study is run with maestro run ... and monitored with maestro status .... Job outputs are organised in directories named automatically after the study name and the submission date and time. The study description and parameters are also copied alongside the results to ease organisation and provenance tracking.

More information: https://maestrowf.readthedocs.io/en/latest/

Generic DAGs

The tools presented in this section differ from the ones in the previous sections in the sense that they manage all dependencies and parametrized jobs by themself without relying onto Slurm specific features. Like Maestro, they will submit then monitor the jobs and take care of ordering, re-submission, logging, etc.

SlurmDagman

SlurmDagman uses the same HTCondor-based workflow description as sdag with additional options and parameters:

JOB dag-node slurm-submission-file
VARS dag-node var-name=var-value [...]
RETRY dag-node max-retries
PARENT parent-dag-node[,...] CHILD child-dag-node[,...]

SlurmDagman also include a tool to translate DagMan options specific to Condor into Slurm lingo.

More information: https://andrestanasijczuk.github.io/SlurmDagman/

Makeflow

Makeflow uses a syntax similar to the GNU Make syntax to describe dependencies. The main difference with the DagMan syntax is that dependencies is not expressed between jobs but rather between data (files). Makeflow then computes the dependences between jobs from the dependences between files.

output: inputs
    command to generate outpus from inputs

Makeflow also ships with a tool to build a visual representation of the workflows. It can also work with a JSON representation of the workflow, which allows to create them programmatically in a easy and simple way.

More information: http://ccl.cse.nd.edu/software/makeflow/

Generic Cyclic workflows

Cycl [https://cylc.github.io/cylc-doc/stable/html/index.html] is a workflow manager specifically designed for cyclic workflows. Jobs in the workflow can be re-submitted until sufficient work has been done.

The workflow description language is a bit more complex. The example below builds the same diamond-shaped workflow as previously, with the additional rule that when D4 is finished, a new cycle starting at A1 can begin.

[scheduler]
    allow implicit tasks = True
[scheduling]
    cycling mode = integer
    initial cycle point = 1
    final cycle point = 10
    [[graph]]
        P1 = """
            A1 => B2 & C3
            B2 & C3 => D4
            D4[-P1] => A1
        """

A workflow is started with cylc play, possibly with the specification of an initial start cycle point as cylc play --start-cycle-point=5.

Full-featured systems

These systems tend to be used by entire teams and require more organisation and maintenance than the tools presented until now. They offer a centralised data storage and workflow sharing system. They were often developed by a specific community and hence are more suited for those communities.

Name | Website | Domain
Fireworks | https://materialsproject.github.io/fireworks/ | Material science
Nextflow | https://nextflow.io | Bioinformatics
Merlin | https://merlin.readthedocs.io/ | Machine learning
DAGMan | https://htcondor.readthedocs.io/ | Particle physics
Pegasus | https://pegasus.isi.edu/ | Agnostic

Further reading

References:

Curcin, Vasa & Ghanem, Moustafa. (2009). Scientific workflow systems - Can one size fit all?. Cairo Int Biomed Eng Conf. 2008. 1 - 9. 10.1109/CIBEC.2008.4786077

Deelman, Ewa & Gannon, Dennis & Shields, Matthew & Taylor, Ian. (2009). Workflows and e- Science: An overview of workflow system features and capabilities. Future Generation Computer Systems. 25. 524-540. 10.1016/j.future.2008.06.012.

Liu, Ji & Pacitti, Esther & Valduriez, Patrick & Mattoso, Marta. (2015). A Survey of Data-Intensive Scientific Workflow Management. Journal of Grid Computing. 13. 10.1007/s10723-015-9329-8.

Badia, Rosa M. & Ayguade, E. & Labarta, Jesús. (2017). Workflows for science: a challenge when facing the convergence of HPC and Big Data. Supercomputing Frontiers and Innovations. 4. 27-47. 10.14529/jsfi170102.

Ferreira da Silva, Rafael & Filgueira, Rosa & Pietri, Ilia & Jiang, Ming & Sakellariou, Rizos & Deelman, Ewa. (2017). A characterization of workflow management systems for extreme-scale applications. Future Generation Computer Systems. 75. 10.1016/j.future.2017.02.026.

Deelman, Ewa & Peterka, Tom & Altintas, Ilkay & Carothers, Christopher & Dam, Kerstin & Moreland, Kenneth & Parashar, Manish & Ramakrishnan, Lavanya & Taufer, Michela & Vetter, Jeffrey. (2017). The future of scientific workflows. The International Journal of High Performance Computing Applications. 32. 109434201770489. 10.1177/1094342017704893.

Lists:

An incomplete list (of more than 300 entries...) of workflow management systems: https://github.com/common-workflow-language/common-workflow-language/wiki/Existing-Workflow-systems

Another list: http://meirwah.github.io/awesome-workflow-engines/

Yet another list: https://github.com/pditommaso/awesome-pipeline

Communities:

https://workflows.community

https://www.commonwl.org

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Slurm interactive jobs

When submitting an interactive job to Slurm, there are several issues to take into account:

the interactive session might start when you are not available ;

your personnal computer might get disconnected.

Planning the start of the job

While there is not way, besides creating an advance reservation [https://slurm.schedmd.com/reservations.html], to ensure a job starts at a specific wanted time, you can make sure it does not start at an unwanted time. For instance, you might not be interested for a job to start at 2:00 AM. Or it is 4PM and you leave in one hour and you are not interesting for the job to start with so little time left.

When you submit the job, you can specify the earliest time at which the job should start, and Slurm will not consider that job until that specified time has come. That is done with the --begin option common to sbatch, srun and salloc. Here are some examples:

--begin=2023-03-12T08:30
--begin=tomorrowT09:00
--begin=now+5daysT09:30:00

Make sure to specify a start time in addition to the date, because the default is 00:00:00.

Once the job has been submitted, you can alter the start time if you changed you mind with the scontrol command:

scontrol update jobid=<JOBID> StartTime=now+5daysT09:30:00

And you can remove the constraint with

scontrol update jobid=<JOBID> StartTime=now

See the full documentation for --begin in the Slurm documentation [https://slurm.schedmd.com/sbatch.html#OPT_begin]

Preventing disconnections

Interactive sessions are typically requested with srun --pty bash, or with the salloc command. Both commands will block until Slurm allocates the resources, and will not survive an SSH disconnection from the login node.

Start srun or salloc in a Tmux session

To make sure the salloc or srun keep waiting for the job to start even if your laptop/desktop is disconnected, put to sleep, or abruptly reboots, you need to use a terminal multiplexer [https://en.wikipedia.org/wiki/Terminal_multiplexer]. The most used terminal multiplexers are GNU screen and Tmux. This document will focus on Tmux as it offers more feature than Screen while at the same time being nearly as commonly installed.

The basic idea is to start the tmux command as soon as you are connected with SSH. You will see the screen clear up, and you will enter a tmux session. From there, you can work as usual, and if you happen to be disconnected, you can SSH back to the login node and run the tmux attach command to re-attach to the running Tmux session that has survived the disconnection.

So the basic idea is to start a Tmux session, and inside that session, run the srun or salloc command. Once the command has started and is waiting for Slurm to allocate the resources, you can detach from it if you want by pressing (and release) CTRL-b and then hitting d “blindly”. This will bring yo back to the initial shell session you got when you connected with SSH.

Tmux session can have names so you can start multiple ones and attach/detach from them at will. For a more comprehensive tutorial on the features of Tmux, you can refer to this document [https://www.redhat.com/sysadmin/introduction-tmux-linux].

Start tmux in a sbatch submission script

The above paragraph explained how to start a Tmux session on the login node to immunize the srun or salloc command from SSH disconnections.

Another option, which is sligly more complicated to setup, but offers more flexibility, is to start Tmux in a (non-interactive) sbatch session and attach to it when the job has started.

This allows to

have commands starting before tmux is started, or in parallel ;

have an interactive job that survives login node reboots or other problems ;

have split panes in the Tmux session that are all running on the compute node ;

start commands inside Tmux automatically upon job start.

Tmux must be started in “detached” mode from the beginning, otherwise it will complain “open terminal failed: not a terminal”. It is also a good idea to name the session in case you have multiple jobs running on the same compute node.

An example submission script can be:

#!/bin/bash
#SBATCH ... # some job options

# Commands to start outside of the tmux session
# Append an `&` sign to make the command run in parallel
# to the tmux session
./prepare-job.sh

# Start tmux
tmux new -d -s "$SLURM_JOB_ID"

# Run tmux commands to setup the session
tmux split-window -h # this will setup two panes, splitting the screen horizontally

# Run commands inside the Tmux session
tmux send-keys -t "$SLURM_JOB_ID" "echo hello world" Enter

# Finally, make job wait for user to connected
sleep 8h

Once the job has started, you can use srun to attach to the Tmux session:

srun --jobid <JOBID> --pty tmux a -t <JOBID>

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

Matlab

Matlab is not installed.

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

R

Installing a R package on the clusters

Created using Sphinx 1.4.1.

Navigation

index

next |

previous |

CÉCI »

R »

Installing a R package on the clusters

To install a R package, you need to use the install.packages() command. By default, it will try to save the installed package in the global library where a regular (non-root) user cannot write.

Using a local library

As R notes that it cannot write, it will ask whether you want to create a local library. Simply answer ‘y’ to that question to get started.

ceciuser@cecicluster:~ $ R

R version 2.13.1 (2011-07-08)
Copyright (C) 2011 The R Foundation for Statistical Computing
ISBN 3-900051-07-0
Platform: x86_64-redhat-linux-gnu (64-bit)
[...]

The .libPaths() command lists the places where R will search for libraries, and use the first item of the list as target for new package installs.

> .libPaths()
[1] "/usr/lib64/R/library" "/usr/share/R/library"

Let’s install the dummy package

> install.packages('dummy')

R tries to install it in the global library

Installing package(s) into ‘/usr/lib64/R/library’
(as ‘lib’ is unspecified)
Warning in install.packages("dummy") :
  'lib = "/usr/lib64/R/library"' is not writable

but quickly notes that it cannot write in the global place, and asks whether it should create a local library. Simply answer ‘y’.

Would you like to create a personal library
'~/R/x86_64-redhat-linux-gnu-library/2.13'
to install packages into?  (y/n) y

and the installation process resumes as usual.

--- Please select a CRAN mirror for use in this session ---
Loading Tcl/Tk interface ... done
trying URL 'https://www.freestatistics.org/cran/src/contrib/dummy_0.1.3.tar.gz'
Content type 'application/x-gzip' length 3710 bytes
opened URL
==================================================
downloaded 3710 bytes

* installing *source* package ‘dummy’ ...
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
  converting help for package ‘dummy’
    finding HTML links ... done
    categories                              html
    dummy                                   html
    dummyNews                               html
** building package indices ...
** testing if installed package can be loaded

* DONE (dummy)

The downloaded packages are in
  ‘/mnt/fhgfs/RtmpgDN1Y8/downloaded_packages’
> q()
Save workspace image? [y/n/c]: n

After that, the new package is installed in the user’s home directory:

ceciuser@cecicluster:~ $ ls R/x86_64-redhat-linux-gnu-library/2.13/dummy/
DESCRIPTION  help  html  INDEX  Meta  NAMESPACE  NEWS  R

Note that R saves the configuration for future uses:

ceciuser@cecicluster:~ $ R

[...]

> .libPaths()
[1] "/home/users/d/f/ceciuser/R/x86_64-redhat-linux-gnu-library/2.13"
[2] "/usr/lib64/R/library"
[3] "/usr/share/R/library"

The libPaths() command now lists the home directory as the first item. Therefore, subsequently installed packages will be saved in the home directory.

> library(dummy)
dummy 0.1.3
dummyNews()
> install.packages('test')
Installing package(s) into ‘/home/users/d/f/ceciuser/R/x86_64-redhat-linux-gnu-library/2.13’
(as ‘lib’ is unspecified)
--- Please select a CRAN mirror for use in this session ---
Loading Tcl/Tk interface ... done
[...]

Final remarks

Computer-nodes can have different architectures , if you install the package in the font-end of a cluster [https://www.ceci-hpc.be/clusters.html] with multiple architectures, you need to use --constraint option in your submission script to use only the set of computes nodes with the same architecture than the front-end.

You can see the front-end architecture with the command:

lscpu | grep "Model name"

To see the compute nodes architecture use the command:

info -o "%20N %20P %b"

If the package you want to install has operating system-level dependencies (i.e. the package depends on core libraries), then you will need to ask the system administrator to install the package.

If you feel that the package you want to install is useful for many users, you can also ask a system administrator to install it globally.

Finally, note that packages depend on the R version you are using. Make sure to load the right module before you install a package!

Created using Sphinx 1.4.1.

Navigation

index

previous |

CÉCI »

Python

On the CÉCI clusters, the Python interpreter is installed, like the other software tools, with EasyBuild. EasyBuild allows installing multiple versions of Python, which can be loaded through “modules” with the ml <module name> command. The list of Python modules can be obtained with ml spider Python:

[dfr@lemaitre3 ~]$ ml spider Python

-------------------------------------------------------------------------------------------------------
  Python:
-------------------------------------------------------------------------------------------------------
    Description:
      Python is a programming language that lets you work more quickly and integrate your systems more
      effectively.

     Versions:
        Python/2.7.11-intel-2016a
        Python/2.7.12-foss-2016b
[...]
        Python/3.9.6-GCCcore-11.2.0-bare
        Python/3.9.6-GCCcore-11.2.0

-------------------------------------------------------------------------------------------------------
  To find other possible module matches execute:

      $ module -r spider '.*Python.*'

-------------------------------------------------------------------------------------------------------
  For detailed information about a specific "Python" package (including how to load the modules) use the mo dule's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider Python/3.9.6-GCCcore-11.2.0
-------------------------------------------------------------------------------------------------------

Having chosen a specific module, you will need to find out in which release it is available, with the more specific ml spider <full module name> command:

[dfr@lemaitre3 ~]$ ml spider Python/3.9.6-GCCcore-11.2.0

-------------------------------------------------------------------------------------------------------
  Python: Python/3.9.6-GCCcore-11.2.0
-------------------------------------------------------------------------------------------------------
    Description:
      Python is a programming language that lets you work more quickly and integrate your systems more
      effectively.

    You will need to load all module(s) on any one of the lines below before the "Python/3.9.6-GCCcore-11.2.0" module is available to load.

      releases/2021b

[...]

Warning

Make sure to understand the module systems and the concept of releases before reading on. They are explained in the Using pre-installed software section.

Note

When looking you are loading and unloading modules to test them, you can use the ml purge command to reinitialise the environment and remove all loaded modules.

Scientific Python and the SciPy-bundle

Python is a generic programming language ; to be useful for scientific computing, its library must be extended through packages. The most used and known Python packages that are necessary for scientific computing are numpy and scipy.

Once a Python module is loaded, the packages that are installed in that specific Python module can be used. They can be listed with the pip freeze command:

[dfr@lemaitre3 ~]$ pip freeze
alabaster @ file:///usr/local[...]/alabaster/alabaster-0.7.12
appdirs @ file:///usr/local[...]/appdirs/appdirs-1.4.4
asn1crypto @ file:///usr/local[...]/asn1crypto/asn1crypto-1.4.0
atomicwrites @ file:///usr/local[...]/atomicwrites/atomicwrites-1.4.0
attrs @ file:///usr/local[...]/attrs/attrs-21.2.0
Babel @ file:///usr/local[...]/Babel/Babel-2.9.1
bcrypt @ file:///usr/local[...]/bcrypt/bcrypt-3.2.0
bitstring @ file:///usr/local[...]/bitstring/bitstring-3.1.7
blist @ file:///usr/local[...]/blist/blist-1.3.6
CacheControl @ file:///usr/local[...]/CacheControl/CacheControl-0.12.6
[...]

Note

In the older Python modules, Numpy and SciPy are part of the Python module so loading the Pyton module automatically enables them. On newer Python modules, this is not the case anymore.

In the recent EasyBuild releases, the Python module does not contain scientific packages. Rather, a specific extension module is available, named SciPy-bundle, which can be loaded directly to have a Python environment with Numpy, Scipy, Pandas, etc. That module will automatically load the Python module on which it was built.

[dfr@lemaitre3 ~]$ ml spider SciPy-bundle

-------------------------------------------------------------------------------------------------------
  SciPy-bundle:
-------------------------------------------------------------------------------------------------------
    Description:
      Bundle of Python packages for scientific software

     Versions:
        SciPy-bundle/2019.03-foss-2019a
        SciPy-bundle/2019.10-foss-2019b-Python-2.7.16
        SciPy-bundle/2019.10-foss-2019b-Python-3.7.4
[...]
        SciPy-bundle/2021.05-foss-2021a
        SciPy-bundle/2021.10-foss-2021b-Python-2.7.18
        SciPy-bundle/2021.10-foss-2021b
        SciPy-bundle/2021.10-intel-2021b

-------------------------------------------------------------------------------------------------------
  For detailed information about a specific "SciPy-bundle" package (including how to load the modules) use
the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider SciPy-bundle/2021.10-intel-2021b
-------------------------------------------------------------------------------------------------------

The Python packages that the SciPy-bundle module offers can be listed with the ml help SciPy-bundle command:

[dfr@lemaitre3 ~]$ ml help SciPy-bundle

----------------------- Module Specific Help for "SciPy-bundle/2021.05-foss-2021a" ------------------------

Description
===========
Bundle of Python packages for scientific software

More information
================
 - Homepage: https://python.org/

Included extensions
===================
Bottleneck-1.3.2, deap-1.3.1, mpi4py-3.0.3, mpmath-1.2.1, numexpr-2.7.3,
numpy-1.20.3, pandas-1.2.4, scipy-1.6.3

and can be verified of course with the pip command.

[dfr@lemaitre3 ~]$ pip freeze | grep SciPybundle
Bottleneck @ file:///usr/local[...]/foss-2021a/Bottleneck/Bottleneck-1.3.2
deap @ file:///usr/local[...]/foss-2021a/deap/deap-1.3.1
mpi4py @ file:///usr/local[...]/foss-2021a/mpi4py/mpi4py-3.0.3
mpmath @ file:///usr/local[...]/foss-2021a/mpmath/mpmath-1.2.1
numexpr @ file:///usr/local[...]/foss-2021a/numexpr/numexpr-2.7.3
numpy @ file:///usr/local[...]/foss-2021a/numpy/numpy-1.20.3
pandas @ file:///usr/local[...]/foss-2021a/pandas/pandas-1.2.4
scipy @ file:///usr/local[...]/foss-2021a/scipy/scipy-1.6.3

Note

Some specific Python packages can have their own extension module

Extension-only modules

Additionally, some specific Python packages such as for instance TensorFlow, or Matplotlib are made available as extension modules, which allows even to have multiple versions of TensorFlow available for the same version of Python.

[dfr@lemaitre3 ~]$ ml spider Tensorflow

-------------------------------------------------------------------------------------------------------
  TensorFlow:
-------------------------------------------------------------------------------------------------------
    Description:
      An open-source software library for Machine Intelligence

     Versions:
        TensorFlow/1.12.0-foss-2018b-Python-3.6.6
        TensorFlow/1.15.2-foss-2019b-Python-3.7.4
        TensorFlow/2.2.0-foss-2019b-Python-3.7.4
        TensorFlow/2.4.1-foss-2020b

-------------------------------------------------------------------------------------------------------
  For detailed information about a specific "TensorFlow" package (including how to load the modules) use th
e module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider TensorFlow/2.4.1-foss-2020b
-------------------------------------------------------------------------------------------------------

Such extension modules include the packages on which the main package depends:

[dfr@lemaitre3 ~]$ ml help TensorFlow

------------------------- Module Specific Help for "TensorFlow/2.4.1-foss-2020b" --------------------------

Description
===========
An open-source software library for Machine Intelligence

More information
================
 - Homepage: https://www.tensorflow.org/

Included extensions
===================
absl-py-0.10.0, astunparse-1.6.3, cachetools-4.2.0, dill-0.3.3, gast-0.3.3,
google-auth-1.24.0, google-auth-oauthlib-0.4.2, google-pasta-0.2.0,
grpcio-1.32.0, gviz-api-1.9.0, h5py-2.10.0, Keras-Preprocessing-1.1.2,
Markdown-3.3.3, oauthlib-3.1.0, opt-einsum-3.3.0, portpicker-1.3.1,
pyasn1-modules-0.2.8, requests-oauthlib-1.3.0, rsa-4.7, tblib-1.7.0,
tensorboard-2.4.1, tensorboard-plugin-profile-2.4.0, tensorboard-plugin-
wit-1.8.0, TensorFlow-2.4.1, tensorflow-estimator-2.4.0, termcolor-1.1.0,
Werkzeug-1.0.1, wrapt-1.12.1

It will furthermore load the modules that are necessary, among which, the``Python`` and SciPy-bundle modules.

User-installed packages

If you cannot find the package you need in the ones provided by the modules, you can install them by yourself as explained in the Installing languages extensions section.

Created using Sphinx 1.4.1.

Navigation

index

CÉCI »

Index

A | B | C | D | E | F | G | H | I | J | M | O | P | Q | R | S | T | W

A

Account

B

Backfill

C

Cache

CLI

Compiler

Compute node

Console

Core

CPU

CPU.hour

D

Debugger

Directory

Disk

Distributed-memory job

Distributed-memory programming

E

Environment module

Environment Variables

F

Fairshare

File

File permission

Filesystem

G

GPU

H

Hardware thread

HPC cluster

I

Interpreter

J

Job

Job priority

Job scheduler

Job submission script

M

MPI

O

OpenMP

Operating system

Optimised library

P

Parallel job

Process

Processor

Profiler

Program

Q

Quality of service

R

RAM

Release

Resource requirements

S

Scalability

Scheduling

Serial job

Share

Shared-memory job

Shared-memory programming

Shell

Software thread

SSH

Step

T

Task

Terminal

Toolchain

W

Wall time

Created using Sphinx 1.4.1.