Skip to content

Job examples using the different filesystems

The location where your job reads and writes its data has a large impact on its speed and on the other users of the cluster. This page gives quick sample submission scripts using the filesystems available on the CÉCI clusters the right way, depending on the type of job. If you are not familiar with job submission yet, start with Submitting jobs with Slurm.

The rule of thumb is:

  • single-node job (serial, OpenMP, Python multiprocessing...) whose data fit in the node: use the local scratch, referenced by the $LOCALSCRATCH environment variable. If the data do not fit, use the global scratch.
  • multi-node job (e.g. MPI), or data to pass from one job to the next: use the global scratch, referenced by the $GLOBALSCRATCH environment variable.
  • use your home $HOME to store your programs and submission scripts, but not as the working directory of your jobs. Reading small input files from it or copying small results back to it is fine.

Single-node job using the local scratch

The Slurm configuration on the CÉCI clusters creates a private $LOCALSCRATCH directory on the compute node when the job starts, and deletes it when the job ends.

The example below is for a hypothetical serial Python program that reads an input file input.in and writes an output file output.out. In the directory from which you submit the job, you have prepared the input of a specific case, named input_case1.in. The script copies it to the local scratch, runs the program there, and copies the result back to the submission directory:

#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=res.txt
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time=15:00
#SBATCH --mem-per-cpu=2000

module load SciPy-bundle

# 1. copy the input file to the local scratch and move there
cp ${SLURM_SUBMIT_DIR}/input_case1.in ${LOCALSCRATCH}/input.in
cd ${LOCALSCRATCH}

# 2. run the program: it stays in $HOME, only its I/O is on the local scratch
srun python ${HOME}/mycode/mycode.py

# 3. copy the results back to the submission directory before the end of the job
cp output.out ${SLURM_SUBMIT_DIR}/

$SLURM_SUBMIT_DIR is set by Slurm to the directory from which you ran sbatch myjob.sh. In this example, the input file input_case1.in must therefore be in that directory also.

The same pattern works for multithreaded jobs (OpenMP, Python multiprocessing...): set for instance #SBATCH --cpus-per-task=4 in the script header and, for OpenMP, add export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK before running the program. All the threads run on the same node and share the same $LOCALSCRATCH.

Warning

Everything left in $LOCALSCRATCH is lost when the job ends. If the job is killed (for instance when it reaches its time limit) before reaching the cp commands, the results are lost.

Single-node job using the global scratch

If the input and output data are too big for the local scratch, you have to use the global scratch. It is visible from the login nodes and from all the compute nodes, so you can prepare the input of your job there with nothing to copy around. Create a directory and move inside, for instance:

$ mkdir -p $GLOBALSCRATCH/myjob
$ cd $GLOBALSCRATCH/myjob

Then either download the input data to work on, or transfer it as explained in the Copying files section. Put the input file input.in and the submission script in this directory, and submit the job from there:

#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=res.txt
#SBATCH --ntasks=1
#SBATCH --time=15:00
#SBATCH --mem-per-cpu=2000

module load SciPy-bundle

# the job starts in the directory it was submitted from on $GLOBALSCRATCH
# the input is already there, and the output will be written there too
srun python ${HOME}/mycode/mycode.py
$ cd $GLOBALSCRATCH/myjob
$ sbatch myjob.sh

As the results stay on the global scratch, a following job can use them directly.

A good practice is to put a reference to the case in the name of the directory, e.g. $GLOBALSCRATCH/myjob/run_X_Y_Z/, with its input.in inside. As the job starts in the directory it is submitted from, its output files are written next to the input, so each run keeps all its files together and is easy to find again. You can also prepare at once the input of several different cases, each in its own run_X_Y_Z directory and launch several independent jobs.

Big input files but small output

A special case is a job which needs big input files, that must stay on the global scratch, but produces small output files, which can be written on the local scratch. Submit the job from a directory on the global scratch, where the big input is. The output must of course fit in the local scratch.

If your program has an option to choose its output directory, such as --output-dir, set it to $LOCALSCRATCH:

#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=res.txt
#
#SBATCH --ntasks=1
#SBATCH --time=15:00
#SBATCH --mem-per-cpu=2000

module load SciPy-bundle

# the job starts in the directory it was submitted from, on $GLOBALSCRATCH:
# the big input is read from there, but the output is written on the local scratch
srun python ${HOME}/mycode/mycode.py --output-dir=${LOCALSCRATCH}

# copy the useful output back before the end of the job
cp ${LOCALSCRATCH}/output.out ${SLURM_SUBMIT_DIR}/

If your program reads its input from a path given as an argument and writes its output in the current directory, cd to the local scratch before the run and pass the path of the input in the submission directory:

#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=res.txt
#
#SBATCH --ntasks=1
#SBATCH --time=15:00
#SBATCH --mem-per-cpu=2000

module load SciPy-bundle

# work in the local scratch, the big input stays on the global scratch
cd ${LOCALSCRATCH}
srun python ${HOME}/mycode/mycode.py ${SLURM_SUBMIT_DIR}/big_input.dat

# copy the useful output back before the end of the job
cp output.out ${SLURM_SUBMIT_DIR}/

In both cases, whatever your program writes in $LOCALSCRATCH (the output, but also any log or intermediate file) is lost when the job ends: copy back everything you need before the end of the job.

Multi-node job

The local scratch of a node is not visible from the other nodes. When the tasks of a job running on several nodes need to read or write the same files, the global scratch must be used:

#!/bin/bash
#
#SBATCH --job-name=test_mpi
#SBATCH --output=res_mpi.txt
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --time=60:00
#SBATCH --mem-per-cpu=1000

module load OpenMPI

# one directory per job on the global scratch, visible from all the nodes
RUNDIR=${GLOBALSCRATCH}/mpijob_${SLURM_JOB_ID}
mkdir -p ${RUNDIR}
cp ${SLURM_SUBMIT_DIR}/input.in ${RUNDIR}/
cd ${RUNDIR}

srun ${HOME}/my_mpi_program/run.x input.in

The program is the one you would run in any MPI job (see the MPI example); only the working directory changes. The results stay in $RUNDIR: copy them wherever you need, or leave them there for a following job. Remove the files you no longer need.

Note

If several tasks write files independently, make sure they never write to the same file: include for instance the MPI rank, or $SLURM_PROCID, in the file names. This does not apply if a single process (e.g. rank 0) does all the I/O, or if the tasks write through a parallel I/O library such as MPI-IO.

A local scratch directory also exists on each node allocated to a multi-node job. If you want to use it, the input files must be copied on each node; see F.A.Q.-Q11 for how to do it.

Job array

Each element of a job array (an array task) is a separate job: it gets its own $LOCALSCRATCH on whatever node it runs, so array tasks never clash there. In this example, the inputs and the results are kept on the global scratch, where all the array tasks can see them, but the I/O is performed on the local scratch:

#!/bin/bash
#
#SBATCH --job-name=test_array
#SBATCH --output=res_array_%a.txt
#
#SBATCH --ntasks=1
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=100
#
#SBATCH --array=0-7

FILES=("${GLOBALSCRATCH}"/data/*)

# copy the input file of this array task to its own local scratch
cp "${FILES[$SLURM_ARRAY_TASK_ID]}" "${LOCALSCRATCH}/input.dat"
cd ${LOCALSCRATCH}

srun ${HOME}/mycode/my_program.exe input.dat output.dat

# copy the result back, with a different name for each array task
mkdir -p ${GLOBALSCRATCH}/results
cp output.dat ${GLOBALSCRATCH}/results/output_${SLURM_ARRAY_TASK_ID}.dat

The array range must match the number of files in the input directory, as explained in the job array example.

Good practices

  • Keep your programs and scripts in $HOME, but do not use it as the working directory of your jobs.
  • It is also fine to keep programs and scripts in $GLOBALSCRATCH, but keep a master copy elsewhere as it can be purged
  • For single-node jobs, prefer $LOCALSCRATCH for the I/O whenever the data fits there: the I/O will run faster and you will avoid unnecessary load on the shared network filesystems
  • For multi-node jobs always use $GLOBALSCRATCH for input and output files
  • Clean the global scratch when you do not need your files anymore
  • None of the CÉCI cluster filesystems is backed up or meant for long-term storage: keep a copy of what matters elsewhere