Jupyter¶

For heavy computations, you can run a Jupyterhub service in a Slurm job allocation on any cluster. Jupyterhub allows you, among others, to access a to a python notebook. To achieve this, you need to the following steps:

Be able to ssh to the clusters with a native SSH client in Linux/Mac or Windows
Find the module to load
Submit a jupyter job in a cluster
Connect your browser to the JupyterLab web instance

Find the module to load¶

Connect to a CECI cluster and look for a JupyterLab module:

$ module spider JupyterLab

Chose one of the available modules and use module spider again to get the module dependencies. Example:

module spider JupyterLab/4.0.5-GCCcore-12.3.0
...
You will need to load all module(s) on any one of the lines below before the "JupyterLab/4.0.5-GCCcore-12.3.0" module is available to load.

  releases/2023a
...

In this case you will need to load releases/2023a before loading IPython.

Submit a job to start jupyter¶

Once you know which module to load, submit a job to work for 1 hour with the command:

srun -t 1:00:00 --pty bash -c 'ml releases/2023a ; ml JupyterLab; jupyter lab --ip $(hostname -i | sed "s/.* //")'

...
...
[I 2025-05-08 18:28:47.123 LabApp] JupyterLab extension loaded from /opt/sw/arch/easybuild/2023a/software/JupyterLab/4.0.5-GCCcore-12.3.0/lib/python3.11/site-packages/jupyterlab
[I 2025-05-08 18:28:47.123 LabApp] JupyterLab application directory is /opt/sw/arch/easybuild/2023a/software/JupyterLab/4.0.5-GCCcore-12.3.0/share/jupyter/lab
[I 2025-05-08 18:28:47.123 LabApp] Extension Manager is 'pypi'.
[I 2025-05-08 18:28:47.124 ServerApp] jupyterlab | extension was successfully loaded.
[I 2025-05-08 18:28:47.125 ServerApp] Serving notebooks from local directory: /home/users/a/l/alozano
[I 2025-05-08 18:28:47.125 ServerApp] Jupyter Server 2.7.2 is running at:
[I 2025-05-08 18:28:47.125 ServerApp] http://10.0.7.36:8888/lab?token=7ab5095028b310cf7578841fd92dc099b10ccaeba33ecdd2
[I 2025-05-08 18:28:47.125 ServerApp]     http://127.0.0.1:8888/lab?token=7ab5095028b310cf7578841fd92dc099b10ccaeba33ecdd2
[I 2025-05-08 18:28:47.125 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2025-05-08 18:28:47.134 ServerApp]

    To access the server, open this file in a browser:
        file:///home/users/c/e/cecilogin/.local/share/jupyter/runtime/jpserver-565107-open.html
    Or copy and paste one of these URLs:
        http://10.0.7.36:8888/lab?token=7ab50950e8b310c87578841fd92dc099ba0ccaeba33ecdd2
        http://127.0.0.1:8888/lab?token=7ab50950e8b310c87578841fd92dc099ba0ccaeba33ecdd2

You need to keep the compute node IP address (10.0.7.36) the port (8888) and the connection token for next step. Leave this terminal open until you have finish working. To end the job you can press CTRL-C.

Connect to the JupyterLab web interface¶

Linux or MacOS¶

If you are using Linux or MacOS, then use the nice little tool named Sshuttle tool to access it.

Sshuttle is a program that, according to its documentation, acts as a "Transparent proxy server that works as a poor man's VPN. Forwards over ssh. Doesn't require admin. Works with Linux and MacOS." Installation on a Linux or MacOS laptop can be done with apt, pacman, dnf depending on your Linux distribution, or with brew, MacPort on MacOS, but you can also install it with pip or git clone from GitHub directly. See the GitHub page for details.

After you have installed sshuttle start a tunnel to the worker nodes ip range of the cluster where you launched your JupyterLab job. If for example it was on Lyra, you will have to run on a terminal

sshuttle -r lyra 10.0.7.1/24

The IP range for sshuttle depends on the cluster, in the table below you have the corresponding subnets for each case:

Cluster	IP range
nic5	`10.252.1.0/16`
lemaitre4	`10.4.1.1/16`
dragon2	`10.102.169.0/24`
hercules	`192.168.1.100/23`
lyra	`10.0.7.1/24`

When running the command it will ask for your password in order to elevate privileges (sudo). That is needed because it will modify temporarily low-level network routing. As long as it runs, the private network of the cluster will be accessible from your laptop, wherever you are connected. Leave that terminal session open for as long as you need access to the Jupyter service in your job.

If your terminal supports it, you can then click on the URL starting with http://10... from the previous step. Otherwise simply copy that URL in your browser and you should see the Jupyterhub interface.

Once you are finished hit CTRL-C in both terminals to stop everything.

See a video with a connexion example on lemaitre3.

Windows¶

If you are using Windows, you must have setup the ssh native client to connect to the cluster from a Powershell terminal.

You can then start a tunnel to the port where JupyterLab is running on the compute node. To do so open a Powershell terminal and type:

ssh -L 8080:10.0.7.36:8888 lyra

Note that you must replace the section after the 8080: by the IP and port from the url http://10.... you got on the previous step. You should get another session on the cluster, leave this terminal opened.

You can now go to a web browser on your laptop and enter the URL http://localhost:8080, you should get a greeting page from JupyterLab asking to introduce a token. You must enter the one from the url you got before, on that example the value 7ab50950e8b.....

Once you are finished you can stop the JupyterLab job with CTRL-C and disconnect from the other session with the tunnel.