Connecting from UNIX-like environment (Linux, Mac OS X, Cygwin/MingW)

This short tutorial describes how to connect to a CÉCI cluster from a terminal application, such as XTerm, Gnome Terminal, Konsole, iTerm, Linux running on Windows Subsytem For Linux, etc.

Choose your favorite terminal application to follow the steps below.

Note

We will use my-ceci-cluster.example.com as a generic dummy address throughout the guide. When trying to follow the steps on your machine replace that by a real remote host of the CÉCI cluster you wish to use:

  • hmem (UCL): hmem.cism.ucl.ac.be
  • Lemaitre3 (UCL): lemaitre3.cism.ucl.ac.be
  • NIC4 (ULiège) : nic4.segi.ulg.ac.be
  • Hercules (UNamur): hercules.ptci.unamur.be
  • Dragon1 (UMons): dragon1.umons.ac.be
  • Vega (ULB): vega.ulb.ac.be

To decide which cluster is better suited for your kind of jobs, please check the details about them in the CÉCI clusters page.

Note

Please note that since the moment you got the email confirming your account is active, you might need to wait up to ~20 minutes maximum until the access is enabled in all the clusters.

1. SSH connection using the CÉCI key

  1. Once you have received your private key by email, store it in a safe location. The most rational place to store it is in your .ssh folder in your home directory, along with other SSH keys you may have. In case the file was downloaded in the Downloads folder in your home, move it with the command:

    mv ~/Downloads/id_rsa.ceci ~/.ssh
    
  2. Change the permissions of the file so that only you can read it:

    chmod 600 ~/.ssh/id_rsa.ceci
    

    If you don’t do this your SSH client will refuse to use the key giving an error which mentions Permisions 0644 for... are too open.

  3. Connect to the cluster you chose with the ssh command:

    ssh -i ~/.ssh/id_rsa.ceci mylogin@my-ceci-cluster.example.com
    

    Make sure to replace mylogin with your actual login you chose when filling the account request form. You are then asked to type in your passphrase (the one you also chose in the account creation request form).

Note

Upon your very first connection to a CÉCI cluster, you will be greeted by a warning such as:

The authenticity of host 'my-ceci-cluster.example.com (XXX.XXX.XX.XX)' can't be established.
RSA key fingerprint is SHA256:aqUPC4C8gkBjgCUMpHt3kzpciSkQcKR2gNOahnbZN3c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ceci-cluster.ac.be,XXX.XXX.XX.XX' (RSA) to the list of known hosts.

This warning is normal, the SSH program warns that it is the first time it sees this computer. To make sure you are actually connecting to the right machine, you should compare the RSA key fingerprint shown in the message with the fingerprint announced on the Access section of the CÉCI cluster page. The fingerprint can be based on hash function MD5 or SHA-256.

If they match, you are safe to proceed and enter yes. That hostname key will be stored and used to check in every subsquent SSH connection that the server is indeed always the same.

  1. If everything worked properly, you should be greeted by a message similar to:

    Last login: Fri Sep 21 12:04:25 2015 from 123.123.222.222
    Welcome to
                 ____  _____  _____   ______  _    _
                |_   \|_   _||_   _|.' ___  || |  | |
                  |   \ | |    | | / .'   \_|| |__| |_
                  | |\ \| |    | | | |       |____   _|
                 _| |_\   |_  _| |_\ `.___.'\    _| |_
                |_____|\____||_____|`.____ .'   |_____|
    
                      ULg/CECI Infiniband cluster
    
    
     128 nodes: 2 x 8-cores E5-2650 @ 2GHz, 64 GB RAM, QDR Infiniband
    
     Contact, support: nicadm@segi.ulg.ac.be , http://www.ulg.ac.be/nic4
    
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    CÉCI clusters: Hmem - Lemaitre2 - Dragon1 - Hercules - Vega - NIC4
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    275/1920 CPUs available (load 85%) - 72 jobs running, 107 pending.
    
       You currently have 0 job running, 0 pending.
       You are using 15GB ( out of 25GB ) in  /home/ulg/ace/alozano
       You are using 114GB ( out of 210GB ) in  /CECI//home/ulg/ace/alozano
    
       Don't know where to start?
                --> http://www.ceci-hpc.be/install_software.html
                --> http://www.ceci-hpc.be/slurm_tutorial.html
    
If you see something like that, you are already connected to the cluster you chose. Please see the following 2. and 3. sections to see how you can ease the connections to the clusters.

2. Create a ssh_config file

You can dramatically shorten the length of the command line used to connect to the clusters by setting your ssh configuration file properly. Assuming your login is mylogin and you are working with the clustername cluster, add the following to your configuration file ~/.ssh/config (create it if it does not exist):

Host clustername
    HostName my-ceci-cluster.example.com
    User mylogin
    ForwardX11 yes
    ForwardAgent yes
    IdentityFile ~/.ssh/id_rsa.ceci

From now on, you will be able to connect to the cluster simply by issuing the following command:

ssh clustername

The FowardX11 option is needed to open any host program in the client display.

The ForwardAgent option is useful if you have an SSH agent running (in the following section it is explained how to setup one), it will forward a connection to your agent on the remote side. Note that, as explained in the SSH manpage, agent forwarding should be enabled only for trusted remote hosts. It is perfectly safe to use it for the CÉCI clusters, but be cautious with other computers.

You can do the same for each cluster you will connect to. Type man ssh_config for a detailed list of options you can set in your configuration file.

A ssh_config wizard is available to generate this config file for all the CÉCI clusters.

3. Using an SSH agent to avoid typing the passphrase at each login

You can further ease the process by using an SSH agent which will remember the passphrase so you do not have to type it each time you issue the SSH command. In any modern Mac OS or Linux distribution is quite certain that you have an agent already running, if the command:

ssh-add -l

returns The agent has no identities or a line like

2048 SHA256:1S30t9SSWSd2PIF0QVAPj1Xhtnj2UYqPw32ZIAu0MkI ~/.ssh/id_rsa.ceci (RSA)

you don’t have anything else to do, your agent is running. Please, check the notes at the bottom of this section if you are still being requested the passphrase at each cluster login.

If the command responds Could not open a connection to your authentication agent, you can start an agent with:

eval $(ssh-agent)

Then, load the CÉCI key with the command:

ssh-add ~/.ssh/id_rsa.ceci

you’ll be prompted once for your passphrase. From now on, in the current terminal, all ssh commands will be handled by the agent and you won’t have to type your passphrase again.

With the agent running, you can use the -A option of ssh to forward your agent from one computer to another (you don’t need this if you are using the ForwardAgent option in a .ssh/config as explained in the section above). This allows you to connect, or copy files, from one cluster to another effortlessly.

You can have an ssh-agent started automatically at login by using password managing software such as Mac OS Keychain, KDE KWallet, Gnome Keyring, Seahorse, etc.

Note

Some software, such as Gnome Keyring, require the public key to be also present in the .ssh directory to enable the login without retyping passphrases. You can generate the public SSH key from the private key (the one you received from the CÉCI by email) with the following command:

ssh-keygen -yf ~/.ssh/id_rsa.ceci > ~/.ssh/id_rsa.ceci.pub

Note

In Mac OS, after Sierra update 10.12.2, in order to automatically unlock your keys these extra options are required on top of your ~/.ssh/config file:

Host *
    AddKeysToAgent yes
    UseKeychain yes

4. Connecting to the cluster from outside the universities network

As the clusters are not accessible from outside the university networks, you will need to either use a VPN or an SSH gateway. Going through an SSH gateway can be entirely transparent provided your client is correctly configured. The CÉCI does not offer a gateway service so you need to refer to your local support team for more information:

  • UCL: you can use gwceci.cism.ucl.ac.be as SSH gateway using a CECI account. See the CISM documentation.
  • ULB: you can use hydra as SSH gateway. See the hydra documentation.
  • UMons: contact the local support team to learn how to setup the VPN.
  • UNamur: you can use hal.unamur.be using your UNamur identity or use the VPN (see UNamur intranet)
  • ULiège: you need to use the VPN service.

Note that you can use any Linux computer that is connected to your university network to which you have SSH access from your home or abroad to act as gateway.

4.1 Using a VPN

If you can setup a VPN on your machine no extra step is required. When enabling the VPN, you should be able to connect with the same ssh command as if you were inside the university network.

4.2 Using a ssh gateway

To use a gateway you need to do a proxy connection with ssh’s ProxyCommand option:

ssh -o 'ProxyCommand ssh mylogin1@gateway.ac.be -W %h:%p' myCECIlogin@my-ceci-cluster.example.com

Replace mylogin1@gateway.ac.be by your university login name and gateway, myCECIlogin by your CÉCI login and my-ceci-cluster.example.com by any CÉCI cluster you want to connect.

You can add a shortcut in your configuration file ~/.ssh/config:

Host gwceci
    HostName gateway.ac.be
    User mylogin1
    ForwardX11 yes
    ForwardAgent yes

Host clustername_via_proxy
    HostName my-ceci-cluster.example.com
    User myCECIlogin
    ForwardX11 yes
    ForwardAgent yes
    IdentityFile ~/.ssh/id_rsa.ceci
    ProxyCommand ssh gwceci -W %h:%p 2> /dev/null

To connect just type:

ssh clustername_via_proxy

It is important to note that you do not need to be outside the university network to test the SSH gateways. Going through a gateway works as well from within than from outside the university network. So be sure to test everything before you leave.

Troubleshooting

Step 1. Make sure that your key file has the correct permissions:

ls -l ~/.ssh/id_rsa.ceci

should output something that starts with

-rw------

Step 2. Make sure your key and your passphrase match by issuing the following command:

ssh-keygen -yf ~/.ssh/id_rsa.ceci

If the output is ‘load failed’ you either have a wrong passphrase or your key file has been corrupted.

Step 3. Make sure you are connected to your university network: the following command

curl -I -k https://login.ceci-hpc.be/init/

should return something starting with

HTTP/1.1 200 OK
Status: 200 OK

Step 4. Make sure the ssh client is asking for a passphrase and not a password. Otherwise, it means you are not correctly instructing your ssh client to use the private key.

Run the ssh command with the -v option. You should find a line stating

debug1: Offering RSA public key: .ssh/id_rsa.ceci

The line immediately following that one should be something like:

debug1: Server accepts key: pkalg ssh-rsa blen 277

otherwise it means the key your are using does not correspond to the one stored on the cluster. If this happens while you just updated or renewed your account, it may indicate that your new key is not fully propagated yet. Try again later.

Frequently encountered problems

Here are some frequent errors you should avoid.

The permissions on your key file are not correct

If, after running ssh clustername, for instance, you see something like:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/home/dfr/.ssh/id_rsa.ceci' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: /home/dfr/.ssh/id_rsa.ceci
dfr@ceci-cluster.ac.be's password:

it means that Permissions 0644 for the /home/dfr/.ssh/id_rsa.ceci file are too open. Change them to 600 as explained in the first section of this document.

You did not specify the correct path to your SSH private key

If, after running ssh, you are being asked for a password directly,

$ ssh clustername
dfr@ceci-cluter.ac.be's password:

it means that your SSH client did not try to use the SSH key. Make sure you either used the -i option or that your ~/.ssh/config file is properly configured and contains no typos.

You used a wrong username or tried to connect before your keys are synchronised

If, after running ssh, you are being asked for a passphrase, then a password,

$ ssh clustername
Enter passphrase for key '/home/dfr/.ssh/id_rsa.ceci':
dfr@ceci-cluster.ac.be's password:

it often means that the user name you are using is not the correct one.

If you just requested or renewed your account, it could also mean that you are using the new private key while it has not been synchronized to that cluster yet (keep in mind that the clusters are not synchronized simultaneously). Wait about 15 minutes and try again.

You have too many SSH keys lying around

If the server responds with

$ ssh clustername
Received disconnect from host: 2: Too many authentication failures for dfr

that means that your client offered the server too many keys to try and access it. To make sure the client uses only the correct key (i.e. the one specified by the IdentityFile option,) you will need to add IdentitiesOnly yes at the top of your ~/.ssh/config file, or pass it as an argument:

$ ssh -o 'IdentitiesOnly yes' clustername

Note that for the SSH-agent to continue working properly when this option is added, you will need the public SSH key to be present in the .ssh directory.

You can generate the public SSH key from the private key (the one you received from the CÉCI with the following command:

ssh-keygen -yf ~/.ssh/id_rsa.ceci > ~/.ssh/id_rsa.ceci.pub