Writing Job Scripts

A basic script

All options actually have short versions (e.g., --job-name can be replaced by -J). The long names are used here for clarity. Not all options in this script are mandatory, but they represent the minimum recommended for clarity.

I use a generic program in all scripts. Replace it with your actual executable. For testing, you can simply use hostname or ls.

I use ibrain as the argument for --partition. You may not have access to it. Replace it with a partition available in the output of sinfo.

#!/bin/sh
#SBATCH --job-name example_script

#SBATCH --ntasks 1
#SBATCH --cpus-per-task 1
#SBATCH --mem 2G

#SBATCH --partition ibrain
#SBATCH --output %x-%j.out
#SBATCH --error %x-%j.out
#SBATCH --hint=nomultithread


module purge
module load gcc/7.3.1

srun program

A few notes on the options:

• The value given to --job-name is used to identify the job when calling squeue.
• A standard Linux console output consists of the output stream and the error stream. When you use sbatch, neither will appear on the console. Instead, they will be redirected to the file(s) specified by the --output and --error options.
• Scripts have internal variables. Here %x will be replaced by the job name, and %j will be replaced by the job ID. The job ID is automatically assigned when calling sbatch and appears when calling squeue.
• The --hint=nomultithread option is discussed further in the Hyperthreading section. If you don’t care about or understand hyperthreading, just leave it there.

Most long versions of options have an associated variable obtained by:

• Capitalizing every letter.
• Replacing the leading -- by SLURM_.
• Replacing the dashes by underscores.

So, in the example above, the variables $SLURM_JOB_NAME, $SLURM_NTASKS, $SLURM_CPUS_PER_TASK (and more) will be defined right after their corresponding option line.

Decomposing the script

A submission script is composed of the following parts:

• A shebang.
• Options.
• Comments.
• Shell commands.
• Run commands.

Shebang

The shebang is always the first line of the script. It specifies which shell interpreter to use for shell commands. Replace sh with whatever shell you want in the following line.

#!/bin/sh

Options

Options are lines that start with:

#SBATCH

They have two purposes:

• The first is to specify resource allocation. In the example script, the combination of --ntasks and --cpus-per-task asks to allocate a single CPU, and the --mem option asks to allocate 2G.

  Note: Allocating a lot of resources does not mean that all of them will be used. If a program is not configured for parallel execution, it will use one CPU even when more are allocated. Start with a documented estimate, run a small test, then inspect actual resource use with the scheduler rather than logging directly into a compute node. For example:

  bash squeue -j <job_id> # queue state sstat -j <job_id>.batch --format=JobID,MaxRSS,AveCPU sacct -j <job_id> --format=JobID,State,Elapsed,MaxRSS,AllocCPUS

  If you need an interactive diagnostic session, request one through the scheduler, for example srun --pty --cpus-per-task=2 --mem=8G --time=01:00:00 bash. Only SSH to a compute node when your own cluster documentation explicitly permits it.

  Other note: In the current configuration, asking for 1 CPU will actually allocate 2, because Slurm allocates by the core, and cores are hyperthreaded. More details in the Hyperthreading section.

• The second purpose is to be passed to the srun command. Unless overwritten when calling srun, all options given with #SBATCH are assumed.

There are many more options:

man sbatch

Any line starting with # but not followed by SBATCH is a comment. If you want to comment an option, use more than one #:

##SBATCH

To avoid confusion between comment and option when using a single #, the comment in my example script uses two #, even though only one is required.

Shell commands

Shell commands are used to set up Linux environment variables. They should not be preceded by srun.

Run commands

Run commands are your actual computations. They should be preceded by srun. The reason is that, as mentioned above, the options given with #SBATCH apply to srun. In truth, when only one task is given with the --ntasks option, omitting srun will not change anything. Put it there anyway, for consistency.

Hyperthreading

Hyperthreading is activated on the nodes, meaning each core has two CPUs. However, this does not double computational power, as using two CPUs of the same core is less efficient than using two CPUs of different cores. With that in mind, here is how Slurm acts (with the number of CPUs you ask for being the product of --ntasks and --cpus-per-task):

• Without the --hint=nomultithread option, asking for an odd number N of CPUs or asking for N+1 CPUs is the same: Slurm allocates the N+1 CPUs of (N+1)/2 cores.
• With the --hint=nomultithread option, asking for N CPUs will allocate the 2N CPUs of N cores.

Note that if you like to use the --mem-per-cpu option instead of the --mem option, the total allocated memory will be based on the number of CPUs actually allocated, not the number of CPUs you asked for. Examples:

• Without the --hint=nomultithread option, the combination of --ntasks 1 --cpus-per-task 1 --mem-per-cpu 1G will allocate 2G.
• The combination of --ntasks 1 --cpus-per-task 2 --mem-per-cpu 1G --hint=nomultithread will allocate 4G.

There is really only one valid use of hyperthreading: when you want to ask for all the CPUs of a single node. In that case, remove the --hint=nomultithread option and allocate everything.

The rest of this documentation assumes we don’t use hyperthreading.

Running things in parallel

Calling the same program multiple times in parallel

#!/bin/sh
#SBATCH --job-name example_script

#SBATCH --ntasks 3
#SBATCH --cpus-per-task 1
#SBATCH --mem 6G

#SBATCH --partition ibrain
#SBATCH --output %x-%j.out
#SBATCH --error %x-%j.out
#SBATCH --hint=nomultithread


module purge
module load gcc/7.3.1

srun program

Recall that the options given with #SBATCH are passed to srun. In this case, a single call to srun program would be equivalent to srun --ntasks 3 program, which would call program three times.

Several steps in parallel

Here is the full script.

#!/bin/sh
#SBATCH --job-name example_script

#SBATCH --ntasks 3
#SBATCH --cpus-per-task 1
#SBATCH --mem 6G

#SBATCH --partition ibrain
#SBATCH --output %x-%j.out
#SBATCH --error %x-%j.out
#SBATCH --hint=nomultithread


module purge
module load gcc/7.3.1

srun --ntasks=1 program0 &
srun --ntasks=1 program1 &
srun --ntasks=1 program2

Do not forget the --ntasks 1 on the srun lines. If you do, program0 will be called three times, and because three tasks are being run and you only allocated three, program1 and program2 will not be executed at the same time, and will have to wait for the previous tasks to end.

Arrays

This is useful when you want to use the same programs multiple times with various arguments. The requirement is that the arguments differ only by an integer number. Here is the script.

#!/bin/sh
#SBATCH --job-name example_script

#SBATCH --array=0-2

#SBATCH --ntasks 1
#SBATCH --cpus-per-task 1
#SBATCH --mem 2G

#SBATCH --partition ibrain
#SBATCH --output %x-%A-%a.out
#SBATCH --error %x-%A-%a.out
#SBATCH --hint=nomultithread


module purge
module load gcc/7.3.1

srun program --arg=$SLURM_ARRAY_TASK_ID

Here, the variable $SLURM_ARRAY_TASK_ID will take values from 0 to 2. The job ID will be $SLURM_JOB_ID, and the array job ID will be $SLURM_ARRAY_JOB_ID. The array task ID will be $SLURM_ARRAY_TASK_ID.

Running an interactive session

srun --pty --job-name interactive --partition ibrain --mem 2G --cpus-per-task 1 bash

This will give you a shell on a compute node. The options are the same as for sbatch. The --pty option is needed to get a pseudo-terminal. The bash at the end specifies the shell to run. You can replace it with sview to get a graphical interface.

Running a GUI

srun --pty --job-name gui --partition ibrain --mem 2G --cpus-per-task 1 --x11 bash

This will give you a shell on a compute node with X11 forwarding enabled. The --x11 option is needed for X11 forwarding. You can then run graphical applications from this shell.

Custom Module

Creating a custom module

If you have a program that you want to make available to others, you can create a custom module for it. This involves creating a module file that defines the environment variables and paths needed to run your program.

Example module file (myprogram/1.0.lua):

help([[This module loads MyProgram version 1.0]])

prepend_path("PATH", "/path/to/myprogram/bin")
prepend_path("LD_LIBRARY_PATH", "/path/to/myprogram/lib")

Place this file in a directory that is part of the MODULEPATH environment variable. You can check your MODULEPATH with module use.

Using a custom module

Once your custom module is created and placed in the correct location, you can load it like any other module:

module load myprogram/1.0

Conda on a Cluster

Installing Conda

To install Conda on the cluster, you can download the Miniconda installer and run it. Choose a location in your home directory where you have write permissions.

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

Follow the prompts during installation. When asked about initializing Conda, you can choose no if you prefer to activate it manually.

Using Conda Environments

After installation, you can create and manage Conda environments. This allows you to isolate different projects and their dependencies.

Create an environment:

conda create --name myenv python=3.8

Activate an environment:

conda activate myenv

Install packages:

conda install numpy pandas

Deactivate an environment:

conda deactivate

Conda in Submission Scripts

To use Conda environments in your Slurm submission scripts, you need to activate the environment before running your program. Make sure to load the Conda module if necessary, or ensure Conda is in your PATH.

#!/bin/sh
#SBATCH --job-name conda_job
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 1
#SBATCH --mem 4G
#SBATCH --partition ibrain
#SBATCH --output %x-%j.out
#SBATCH --error %x-%j.out

# Load Conda if not in PATH
# module load conda

# Activate your Conda environment
source /path/to/your/miniconda3/bin/activate myenv

# Run your Python script or program
srun python my_script.py

# Deactivate environment (optional, but good practice)
conda deactivate

Compression of files

This document will help you to compress your files on HPC as we see we have less space on cluster its better to compress our fastq files and other result files.

Using gzip

gzip is a common compression utility. It replaces the original file with a compressed version (.gz extension).

Compress a file:

gzip my_file.txt

This will create my_file.txt.gz and remove my_file.txt.

Decompress a file:

gunzip my_file.txt.gz

This will restore my_file.txt and remove my_file.txt.gz.

Using tar with gzip (for directories)

To compress entire directories, you typically use tar to archive the directory first, and then gzip to compress the archive. The tar command has options to do both simultaneously.

Compress a directory:

tar -czvf my_directory.tar.gz my_directory/

• -c: Create a new archive.
• -z: Compress the archive with gzip.
• -v: Verbose output (show progress).
• -f: Specify the archive filename.

Decompress a directory:

tar -xzvf my_directory.tar.gz

• -x: Extract files from an archive.
• -z: Decompress with gzip.
• -v: Verbose output.
• -f: Specify the archive filename.

Using bzip2

bzip2 often provides better compression ratios than gzip, but is slower.

Compress a file:

bzip2 my_file.txt

This creates my_file.txt.bz2.

Decompress a file:

bunzip2 my_file.txt.bz2

Using xz

xz provides even better compression than bzip2 but is the slowest.

Compress a file:

xz my_file.txt

This creates my_file.txt.xz.

Decompress a file:

unxz my_file.txt.xz

Choose the compression method based on your needs for compression ratio versus speed. For large files like fastq, gzip is a common balance. For maximum compression, xz is preferred if time is not critical.