Job examples using the different filesystems¶
The location where your job reads and writes its data has a large impact on its speed and on the other users of the cluster. This page gives quick sample submission scripts using the filesystems available on the CÉCI clusters the right way, depending on the type of job. If you are not familiar with job submission yet, start with Submitting jobs with Slurm.
The rule of thumb is:
- single-node job (serial, OpenMP, Python multiprocessing...) whose data
fit in the node: use the local scratch, referenced by the
$LOCALSCRATCHenvironment variable. If the data do not fit, use the global scratch. - multi-node job (e.g. MPI), or data to pass from one job to the next: use
the global scratch, referenced by the
$GLOBALSCRATCHenvironment variable. - use your home
$HOMEto store your programs and submission scripts, but not as the working directory of your jobs. Reading small input files from it or copying small results back to it is fine.
Single-node job using the local scratch¶
The Slurm configuration on the CÉCI clusters creates a private $LOCALSCRATCH
directory on the compute node when the job starts, and deletes it when the job
ends.
The example below is for a hypothetical serial Python program that reads an
input file input.in and writes an output file output.out. In the directory
from which you submit the job, you have prepared the input of a specific case,
named input_case1.in. The script copies it to the local scratch, runs the
program there, and copies the result back to the submission directory:
#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=res.txt
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time=15:00
#SBATCH --mem-per-cpu=2000
module load SciPy-bundle
# 1. copy the input file to the local scratch and move there
cp ${SLURM_SUBMIT_DIR}/input_case1.in ${LOCALSCRATCH}/input.in
cd ${LOCALSCRATCH}
# 2. run the program: it stays in $HOME, only its I/O is on the local scratch
srun python ${HOME}/mycode/mycode.py
# 3. copy the results back to the submission directory before the end of the job
cp output.out ${SLURM_SUBMIT_DIR}/
$SLURM_SUBMIT_DIR is set by Slurm to the directory from which you ran
sbatch myjob.sh. In this example, the input file input_case1.in must
therefore be in that directory also.
The same pattern works for multithreaded jobs (OpenMP, Python
multiprocessing...): set for instance #SBATCH --cpus-per-task=4 in the script
header and, for OpenMP, add export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
before running the program. All the threads run on the same node and share the
same $LOCALSCRATCH.
Warning
Everything left in $LOCALSCRATCH is lost when the job ends. If the job is
killed (for instance when it reaches its time limit) before reaching the cp
commands, the results are lost.
Single-node job using the global scratch¶
If the input and output data are too big for the local scratch, you have to use the global scratch. It is visible from the login nodes and from all the compute nodes, so you can prepare the input of your job there with nothing to copy around. Create a directory and move inside, for instance:
$ mkdir -p $GLOBALSCRATCH/myjob
$ cd $GLOBALSCRATCH/myjob
Then either download the input data to work on, or transfer it as explained in
the Copying files section.
Put the input file input.in and the submission script in this directory, and
submit the job from there:
#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=res.txt
#SBATCH --ntasks=1
#SBATCH --time=15:00
#SBATCH --mem-per-cpu=2000
module load SciPy-bundle
# the job starts in the directory it was submitted from on $GLOBALSCRATCH
# the input is already there, and the output will be written there too
srun python ${HOME}/mycode/mycode.py
$ cd $GLOBALSCRATCH/myjob
$ sbatch myjob.sh
As the results stay on the global scratch, a following job can use them directly.
A good practice is to put a reference to the case in the name of the directory,
e.g. $GLOBALSCRATCH/myjob/run_X_Y_Z/, with its input.in inside. As the job
starts in the directory it is submitted from, its output files are written next
to the input, so each run keeps all its files together and is easy to find
again. You can also prepare at once the input of several different cases, each
in its own run_X_Y_Z directory and launch several independent jobs.
Big input files but small output¶
A special case is a job which needs big input files, that must stay on the global scratch, but produces small output files, which can be written on the local scratch. Submit the job from a directory on the global scratch, where the big input is. The output must of course fit in the local scratch.
If your program has an option to choose its output directory, such as
--output-dir, set it to $LOCALSCRATCH:
#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=res.txt
#
#SBATCH --ntasks=1
#SBATCH --time=15:00
#SBATCH --mem-per-cpu=2000
module load SciPy-bundle
# the job starts in the directory it was submitted from, on $GLOBALSCRATCH:
# the big input is read from there, but the output is written on the local scratch
srun python ${HOME}/mycode/mycode.py --output-dir=${LOCALSCRATCH}
# copy the useful output back before the end of the job
cp ${LOCALSCRATCH}/output.out ${SLURM_SUBMIT_DIR}/
If your program reads its input from a path given as an argument and writes its
output in the current directory, cd to the local scratch before the run and
pass the path of the input in the submission directory:
#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=res.txt
#
#SBATCH --ntasks=1
#SBATCH --time=15:00
#SBATCH --mem-per-cpu=2000
module load SciPy-bundle
# work in the local scratch, the big input stays on the global scratch
cd ${LOCALSCRATCH}
srun python ${HOME}/mycode/mycode.py ${SLURM_SUBMIT_DIR}/big_input.dat
# copy the useful output back before the end of the job
cp output.out ${SLURM_SUBMIT_DIR}/
In both cases, whatever your program writes in $LOCALSCRATCH (the output, but
also any log or intermediate file) is lost when the job ends: copy back
everything you need before the end of the job.
Multi-node job¶
The local scratch of a node is not visible from the other nodes. When the tasks of a job running on several nodes need to read or write the same files, the global scratch must be used:
#!/bin/bash
#
#SBATCH --job-name=test_mpi
#SBATCH --output=res_mpi.txt
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --time=60:00
#SBATCH --mem-per-cpu=1000
module load OpenMPI
# one directory per job on the global scratch, visible from all the nodes
RUNDIR=${GLOBALSCRATCH}/mpijob_${SLURM_JOB_ID}
mkdir -p ${RUNDIR}
cp ${SLURM_SUBMIT_DIR}/input.in ${RUNDIR}/
cd ${RUNDIR}
srun ${HOME}/my_mpi_program/run.x input.in
The program is the one you would run in any MPI job (see the MPI
example); only the working
directory changes. The results stay in $RUNDIR: copy them wherever you need,
or leave them there for a following job. Remove the files you no longer need.
Note
If several tasks write files independently, make sure they never write to the
same file: include for instance the MPI rank, or $SLURM_PROCID, in the file
names. This does not apply if a single process (e.g. rank 0) does all the I/O,
or if the tasks write through a parallel I/O library such as MPI-IO.
A local scratch directory also exists on each node allocated to a multi-node job. If you want to use it, the input files must be copied on each node; see F.A.Q.-Q11 for how to do it.
Job array¶
Each element of a job array (an array task) is a separate job: it gets its
own $LOCALSCRATCH on whatever node it runs, so array tasks never clash there.
In this example, the inputs and the results are kept on the global scratch,
where all the array tasks can see them, but the I/O is performed on the local
scratch:
#!/bin/bash
#
#SBATCH --job-name=test_array
#SBATCH --output=res_array_%a.txt
#
#SBATCH --ntasks=1
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=100
#
#SBATCH --array=0-7
FILES=("${GLOBALSCRATCH}"/data/*)
# copy the input file of this array task to its own local scratch
cp "${FILES[$SLURM_ARRAY_TASK_ID]}" "${LOCALSCRATCH}/input.dat"
cd ${LOCALSCRATCH}
srun ${HOME}/mycode/my_program.exe input.dat output.dat
# copy the result back, with a different name for each array task
mkdir -p ${GLOBALSCRATCH}/results
cp output.dat ${GLOBALSCRATCH}/results/output_${SLURM_ARRAY_TASK_ID}.dat
The array range must match the number of files in the input directory, as explained in the job array example.
Good practices¶
- Keep your programs and scripts in
$HOME, but do not use it as the working directory of your jobs. - It is also fine to keep programs and scripts in
$GLOBALSCRATCH, but keep a master copy elsewhere as it can be purged - For single-node jobs, prefer
$LOCALSCRATCHfor the I/O whenever the data fits there: the I/O will run faster and you will avoid unnecessary load on the shared network filesystems - For multi-node jobs always use
$GLOBALSCRATCHfor input and output files - Clean the global scratch when you do not need your files anymore
- None of the CÉCI cluster filesystems is backed up or meant for long-term storage: keep a copy of what matters elsewhere