Connecting from UNIX-like environment (Linux, Mac OS X, Cygwin/MingW)

This short tutorial describes how to connect to a CÉCI cluster from a terminal application, such as XTerm, Gnome Terminal, Konsole, iTerm, Linux running on Windows Subsystem For Linux, the native SSH client in Powershell (Windows 10 1809 and higher) etc.

Clusters

All CÉCI users can connect to any of the following clusters:

  • Lemaitre4 (UCL): lemaitre4.cism.ucl.ac.be
  • NIC5 (ULiège) : nic5.uliege.be
  • Hercules2 (UNamur): hercules.ptci.unamur.be
  • Dragon1 (UMons): dragon1.umons.ac.be
  • Dragon2 (UMons): dragon2.umons.ac.be

To decide which cluster is better suited for your kind of jobs, please check the details about them in the CÉCI clusters page.

Gateways

orphan:

As the clusters are not accessible from outside the university networks, you will need to use a SSH gateway and the university VPN in some cases. Going through a SSH gateway can be entirely transparent provided your client is correctly configured. The CÉCI does not offer a centralized gateway service, so you will have to use the one provided by the University you belong to:

  • UCL: use gwceci.cism.ucl.ac.be as SSH gateway with your CÉCI account. See the CISM documentation.
  • ULB: use gwceci.ulb.ac.be as SSH gateway with your CÉCI account. You will need to use a VPN also if you are outside Belgium.
  • UMons: contact the local support team to learn how to setup the VPN, then use dragon2.umons.ac.be as gateway.
  • UNamur: use gwceci.unamur.be with your UNamur identity (eID)
  • ULiège: use gwceci.uliege.be with your CÉCI account AND use the ULiège VPN.

Choose your favorite terminal application to follow the steps below.

Procedure

1. Get the private CÉCI key

To get the private key, you need to create or renew your CÉCI account

Note

Please note that since the moment you got the email with you key confirming your account is active, you might need to wait up to ~20 minutes maximum until the access is enabled in all the clusters.

For some Universities the ceci public key needs to be set in the gateway and this can take some hours.

  1. Once you have received your private key by email, store it in a safe location. The most rational place to store it is in your .ssh folder in your home directory, along with other SSH keys you may have. In case the file was downloaded in the Downloads folder in your home, move it with the command:

    mkdir -p ~/.ssh && mv ~/Downloads/id_rsa.ceci ~/.ssh
    
  2. Change the permissions of the file so that only you can read it:

    chmod 600 ~/.ssh/id_rsa.ceci
    

    If you don’t do this your SSH client will refuse to use the key giving an error which mentions Permisions 0644 for... are too open.

  3. Create your public key:

    ssh-keygen -yf ~/.ssh/id_rsa.ceci > ~/.ssh/id_rsa.ceci.pub
    

    You will be asked by the passphrase you chose on the CÉCI account creation form.

Warning

If for the last step you get an error mentioning either Load key "id_rsa.ceci": invalid format or Load key "id_rsa.ceci": error in libcrypto, it should be fixed by running the command:

dos2unix ~/.ssh/id_rsa.ceci

see here for more information.

2. Connect to a cluster

Even if you are used to use ssh on the command line, the fact of having to go through a gateway makes the command to connect too cumbersome to type it every time. So we will instruct you on how to create a ssh_config file to ease with this.

  1. Go to our ssh_config wizard that will aid in creating the configuration depending on your University.

  2. Choose which is your University and provide the required information.

  3. Use your favorite text editor on the terminal (nano, vim, emacs ...) to create or edit the configuration file ~/.ssh/config e.g.:

    nano ~/.ssh/config
    

    and copy-paste the contents provided by the wizard. If you already have contents there, keep it and paste this block below it, save the changes (ctrl+x then answer yes by pressing y and after that enter). Make sure the file has correct permissions:

    chmod 600 ~/.ssh/config
    
  4. Now everything is set, you can do your first connection using the command:

    ssh cecicluster
    

Where you should replace cecicluster by one of the Host alias for the CÉCI clusters: lemaitre4, hercules, dragon1, dragon2 or nic5.

Note

If it is the first time you use the private key and/or your agent is not running you will be asked for the key passphrase twice. One for the gateway connexion and another for the cluster connexion.

If everything worked properly, you should be greeted by a message similar to

Welcome to


 ██╗     ███████╗███╗   ███╗ █████╗ ██╗████████╗██████╗ ███████╗██████╗
 ██║     ██╔════╝████╗ ████║██╔══██╗██║╚══██╔══╝██╔══██╗██╔════╝╚════██╗
 ██║     █████╗  ██╔████╔██║███████║██║   ██║   ██████╔╝█████╗   █████╔╝
 ██║     ██╔══╝  ██║╚██╔╝██║██╔══██║██║   ██║   ██╔══██╗██╔══╝   ╚═══██╗
 ███████╗███████╗██║ ╚═╝ ██║██║  ██║██║   ██║   ██║  ██║███████╗██████╔╝
 ╚══════╝╚══════╝╚═╝     ╚═╝╚═╝  ╚═╝╚═╝   ╚═╝   ╚═╝  ╚═╝╚══════╝╚═════╝

              Massively parallel CISM-CECI cluster

      80 nodes: 2 x 12-core Intel Skylake 5118@2.3GHz, 96GB RAM
            1:3-blocking OmniPath Architecture network

           contact, support: egs-cism@listes.uclouvain.be
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CÉCI clusters: Hmem - Lemaitre3 - Dragon1 - Dragon2 - Hercules - Vega - NIC4

      You currently have 0 job running, 0 pending.

      Don't know where to start?
               --> https://www.ceci-hpc.be/install_software.html
               --> https://www.ceci-hpc.be/slurm_tutorial.html

Note

Upon your very first connection to each CÉCI cluster, you will be greeted by a warning such as:

The authenticity of host 'my-ceci-cluster.example.com (XXX.XXX.XX.XX)' can't be established.
RSA key fingerprint is SHA256:aqUPC4C8gkBjgCUMpHt3kzpciSkQcKR2gNOahnbZN3c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ceci-cluster.ac.be,XXX.XXX.XX.XX' (RSA) to the list of known hosts.

This warning is normal, the SSH program warns that it is the first time it sees this computer. To make sure you are actually connecting to the right machine, you should compare the RSA key fingerprint shown in the message with the fingerprint announced on the Access section of the CÉCI cluster page. The fingerprint can be based on hash function MD5 or SHA-256.

If they match, you are safe to proceed and enter yes. That hostname key will be stored and used to check in every subsquent SSH connection that the server is indeed always the same.

Note

If you hit an error that mentions something as:

line 12: Bad configuration option: proxyjump

Then change the line with the ProxyJump option in the ~/ssh/.config to read instead:

ProxyCommand ssh gwceci -W %h:%p

3. Using an SSH agent to avoid typing the passphrase at each login

You can further ease the process by using an SSH agent which will remember the passphrase so you do not have to type it each time you issue the SSH command. In any modern Mac OS or Linux distribution is quite certain that you have an agent already running, if the command:

ssh-add -l

returns The agent has no identities or a line like

2048 SHA256:1S30t9SSWSd2PIF0QVAPj1Xhtnj2UYqPw32ZIAu0MkI ~/.ssh/id_rsa.ceci (RSA)

you don’t have anything else to do, your agent is running. Please, check the notes at the bottom of this section if you are still being requested the passphrase at each cluster login.

If the command responds Could not open a connection to your authentication agent, you can start an agent with:

eval $(ssh-agent)

Then, load the CÉCI key with the command:

ssh-add ~/.ssh/id_rsa.ceci

you’ll be prompted once for your passphrase. From now on, in the current terminal, all ssh commands will be handled by the agent and you won’t have to type your passphrase again.

With the agent running, you can use the -A option of ssh to forward your agent from one computer to another (you don’t need this if you are using the ForwardAgent option in a .ssh/config as explained in the section above). This allows you to connect, or copy files, from one cluster to another effortlessly.

You can have an ssh-agent started automatically at login by using password managing software such as Mac OS Keychain, KDE KWallet, Gnome Keyring, Seahorse, etc.

Note

In Mac OS, after Sierra update 10.12.2, in order to automatically unlock your keys these extra options are required on top of your ~/.ssh/config file:

Host *
    AddKeysToAgent yes
    UseKeychain yes

Troubleshooting

Step 1. Make sure that your key file has the correct permissions:

ls -l ~/.ssh/id_rsa.ceci

should output something that starts with

-rw------

Step 2. Make sure your key and your passphrase match by issuing the following command:

ssh-keygen -yf ~/.ssh/id_rsa.ceci

If the output is ‘load failed’ you either have a wrong passphrase or your key file has been corrupted.

Step 4. Make sure the ssh client is asking for a passphrase and not a password. Otherwise, it means you are not correctly instructing your ssh client to use the private key.

Run the ssh command with the -v option. You should find a line stating

debug1: Offering RSA public key: .ssh/id_rsa.ceci

The line immediately following that one should be something like:

debug1: Server accepts key: pkalg ssh-rsa blen 277

otherwise it means the key your are using does not correspond to the one stored on the cluster. If this happens while you just updated or renewed your account, it may indicate that your new key is not fully propagated yet. Try again later.

Frequently encountered problems

Here are some frequent errors you should avoid.

The permissions on your key file are not correct

If, after running ssh cecicluster, for instance, you see something like:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/home/ceciuser/.ssh/id_rsa.ceci' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: /home/ceciuser/.ssh/id_rsa.ceci
ceciuser@ceci-cluster.ac.be's password:

it means that Permissions 0644 for the /home/ceciuser/.ssh/id_rsa.ceci file are too open. Change them to 600 as explained in the first section of this document.

You did not specify the correct path to your SSH private key

If, after running ssh, you are being asked for a password directly,

$ ssh cecicluster
ceciuser@ceci-cluster.ac.be's password:

it means that your SSH client did not try to use the SSH key. Make sure you either used the -i option or that your ~/.ssh/config file is properly configured and contains no typos.

You used a wrong username or tried to connect before your keys are synchronised

If, after running ssh, you are being asked for a passphrase, then a password,

$ ssh cecicluster
Enter passphrase for key '/home/ceciuser/.ssh/id_rsa.ceci':
ceciuser@ceci-cluster.ac.be's password:

it often means that the user name you are using is not the correct one.

If you just requested or renewed your account, it could also mean that you are using the new private key while it has not been synchronized to that cluster yet (keep in mind that the clusters are not synchronized simultaneously). Wait about 15 minutes and try again.

You have too many SSH keys lying around

If the server responds with

$ ssh cecicluster
Received disconnect from host: 2: Too many authentication failures for ceciuser

that means that your client offered the server too many keys to try and access it. To make sure the client uses only the correct key (i.e. the one specified by the IdentityFile option,) you will need to add IdentitiesOnly yes at the top of your ~/.ssh/config file, or pass it as an argument:

$ ssh -o 'IdentitiesOnly yes' cecicluster

Note that for the SSH-agent to continue working properly when this option is added, you will need the public SSH key to be present in the .ssh directory.

You can generate the public SSH key from the private key (the one you received from the CÉCI with the following command:

ssh-keygen -yf ~/.ssh/id_rsa.ceci > ~/.ssh/id_rsa.ceci.pub

You copy/pasted the config file to a word processor

If the SSH client complains that

Bad configuration option: \302\240\302\240\302\240\302\240hostname

you should fix your ~/.ssh/config file.

The spaces in your ~/.ssh/config file are so-called “non breaking spaces” probably inserted there by a word processor.

You can try removing and re-inserting the spaces in the configuration file using a regular text editor or use GNU sed

sed -i .bak 's/\xc2\xa0/ /g' ~/.ssh/config

Make sure to use GNU sed, this will not work with the BSD sed that ships with MacOS. You can use Perl as an alternative:

perl -i.bak -pe 's/\xc2\xa0//g' ~/.ssh/config

Both options will create a backup ~/.ssh/config.bak

The passphrase in GNOME keyring was not properly updated

If the SSH client complains that

sign_and_send_pubkey: signing failed: agent refused operation

it could be that the passphrase stored in the GNOME keyring is not correct. You can re-enter it by first flushing them :

gnome-keyring-daemon -r -d

and then re-enter them:

ssh-askpass

The SSH private key was tampered with by Outlook

If ssh-keygen complains that

Load key "id_rsa.ceci": invalid format

or (more recent versions of OpenSSH)

Load key "id_rsa.ceci": error in libcrypto

It could be because the file was “translated” automatically, when you downloaded it to your computer, to the DOS text file type that uses a different convention for line breaks than UNIX/Linux text files.

You can check that is the case with

cat -e id_rsa.ceci | grep -c "\^M"

If the output is non-zero, your file is indeed in the wrong format. The expected output should be 0. To fix it, use the dos2unix command:

dos2unix id_rsa.ceci

If dos2unix is not installed and you cannot or do not want to install it, check in any of the following command is installed: tr, col, or sed. If any of those command is installed:

Step 1: run the following command

cp id_rsa.ceci id_rsa.ceci.bak

before to make a copy of the downloaded file; and then

Step 2: run either of

col -b < id_rsa.ceci.bak > id_rsa.ceci

or

tr -d '\r' < id_rsa.ceci.bak > id_rsa.ceci

or

sed 's/\r$//' id_rsa.ceci.bak > id_rsa.ceci

depending on what you have available.

Alternatively, if vim is installed, you can also get the same result by opening the file with vim, then typing :set ff=unix; and then saving the file.

The resulting id_rsa.ceci file should then pass the cat -e id_rsa.ceci | grep -c "\^M" test and be accepted by the SSH client.

The MobaXterm version is too old

If MobaXterm complains that it

Couldn't load private key (unrecognised cipher name)

it is probably because the version you have is too old and does recognise the new SSH ciphers that are introduced in the more recent OpenSSH version used since the upgrade of the CÉCI account management system.

Try upgrading MobaXterm to version 23.2 or more recent.