Basic Slurm Commands

Use Slurm commands to inspect the resources your cluster exposes, submit a small batch job, and find out why a job is waiting or failed. Slurm versions and local policies differ; replace example partitions, module names, paths, and resource limits with values documented by your HPC site.

Before you start

This reference is for learners who already know basic shell commands and have an account with SSH access to a Slurm cluster. Work from the login node only for setup and submission, not for compute work. The cluster examples below require your site's current user documentation. They have not been run on a real cluster.

Check the module environment

Sites may provide software through Environment Modules, Lmod, another environment manager, or a preconfigured batch environment. The module command is often a shell function created by shell-specific initialization. A non-interactive batch shell may not initialize it the same way as your interactive login shell. Check what the site documents for batch scripts; do not guess or copy a generic initialization-file path.

if command -v module >/dev/null 2>&1; then
  module avail
  module list
else
  echo "No module command in this shell; check site instructions for batch-shell initialization."
fi

Module names, versions, shell setup, and R executable names are site-specific; Python applies only to the optional alternative below. Do not assume that an example module name exists. If the site documents a module for the batch environment, use that exact name and initialize Modules as directed by the site. A failed module load must stop the job rather than quietly using another R. If the site does not use modules, use an Rscript path only after the site documentation confirms which R installation is available in non-interactive batch jobs.

Inspect partitions and your queue

Use the cluster's own output to discover partition names and node states; an idle node does not guarantee that your account or request is eligible to run there. Replace PARTITION_NAME with a name reported by your site. State and resource fields describe scheduler configuration, not a promise of availability.

sinfo
sinfo --Node --long
sinfo -p PARTITION_NAME

To see your jobs and a compact pending reason, use:

squeue -u "$USER" -o "%.18i %.9P %.30j %.10u %.2t %.10M %.6D %R"

Common compact states include PD (pending), R (running), CG (completing), CD (completed), and F (failed). The %R field shows a reason for pending jobs and, for other states, may show allocated nodes or a failure explanation. A completed job may disappear from squeue; consult accounting when the site provides it.

Try a tiny local synthetic script

Start with the selected R route. It uses base R only and needs no account, scheduler, module system, third-party package, or research data. Save the code as local-smoke.R and run Rscript local-smoke.R on a machine with R installed. It prints the selected Rscript path, R version, hostname, and a predictable arithmetic result.

R · base R (primary route)

cat("Selected Rscript: ", Sys.which("Rscript"), "\n", sep = "")
cat("R version: ", R.version.string, "\n", sep = "")
cat("Hostname: ", Sys.info()[["nodename"]], "\n", sep = "")
cat("Sum 1..100: ", sum(1:100), "\n", sep = "")

Expected output includes the selected executable path, R version, host name, and Sum 1..100: 5050. The path, version, and host vary by machine. This invented fixture is only a local runtime smoke test; it does not test Slurm, a compute node, scientific software, or any research result.

Python · optional alternative

import platform
import sys

print(f"Selected Python: {sys.executable}")
print(f"Python version: {platform.python_version()}")
print(f"Hostname: {platform.node()}")
print(f"Sum 1..100: {sum(range(1, 101))}")

This alternative needs Python 3 and no third-party package. Save it as local-smoke.py and run python3 local-smoke.py. It performs the same arithmetic smoke test; it does not establish anything about the target cluster.

Prepare a batch script for your selected language

R · base R (primary route)

On the cluster, follow site storage guidance and move to a permitted working directory. Create hello.R there with the R snippet above, then save the following as smoke.sbatch. Replace the R module placeholder with the exact module documented by your HPC site. The example requests one node, one task, one CPU, one gigabyte of memory, and five minutes; these are illustrative limits, not portable recommendations.

#!/usr/bin/env bash
#SBATCH --job-name=r-smoke
#SBATCH --output=slurm-%j.out
#SBATCH --error=slurm-%j.err
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time=00:05:00
#SBATCH --mem=1G

set -euo pipefail

# Replace with the exact R module name and batch-shell setup documented by your site.
if ! command -v module >/dev/null 2>&1; then
  echo "module is not initialized in this batch shell; follow site batch-shell instructions." >&2
  exit 127
fi
if ! module load REPLACE_WITH_SITE_DOCUMENTED_R_MODULE; then
  echo "R module load failed; refusing to use an unrelated installation." >&2
  exit 1
fi
if ! rscript_path=$(command -v Rscript); then
  echo "Rscript is not available after the documented module load." >&2
  exit 127
fi
printf 'Selected Rscript: %s\n' "$rscript_path"
"$rscript_path" --version
hostname
"$rscript_path" hello.R

Check whether your site requires a partition, account, quality-of-service, storage path, or other directive, and add only values it documents for your account. Slurm reads #SBATCH directives before executing the script, so keep them above shell commands and do not put shell variables in their values. If your site does not use modules, replace the module block only with a fail-fast check for the site-documented Rscript executable in non-interactive jobs; do not fall back silently between installations. The selected path is printed so the batch environment is inspectable.

Python · optional batch alternative

This is the same synthetic utility using Python 3 instead of R. Create hello.py with the Python snippet above, then save this alternative as python-smoke.sbatch; substitute the exact site-documented Python module. It is not needed for the R route.

#!/usr/bin/env bash
#SBATCH --job-name=python-smoke
#SBATCH --output=slurm-%j.out
#SBATCH --error=slurm-%j.err
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time=00:05:00
#SBATCH --mem=1G

set -euo pipefail
if ! command -v module >/dev/null 2>&1; then
  echo "module is not initialized in this batch shell; follow site batch-shell instructions." >&2
  exit 127
fi
if ! module load REPLACE_WITH_SITE_DOCUMENTED_PYTHON_MODULE; then
  echo "Python module load failed; refusing to use an unrelated interpreter." >&2
  exit 1
fi
python_path=$(command -v python3) || { echo "python3 is unavailable after module load." >&2; exit 127; }
printf 'Selected Python: %s\n' "$python_path"
python3 --version
hostname
python3 hello.py

The optional Python script has the same site-specific partition, account, resource, and batch-shell caveats. A local R or Python smoke test verifies only that local interpreter and arithmetic. Even a successful submitted job confirms only that this tiny synthetic command ran in the observed cluster context; it does not validate a scientific workflow or make a local test evidence of HPC compatibility.

Submit once and keep the returned identifier

Run the primary R submission from the directory containing smoke.sbatch and hello.R. Keep the following commands in the same Bash shell. To submit the optional Python alternative instead, substitute python-smoke.sbatch and its hello.py input. sbatch --parsable returns the identifier assigned by Slurm; a federated or cluster-qualified response may be JOB_ID;CLUSTER. The validation accepts a numeric identifier and optional non-empty cluster suffix, and rejects unexpected output instead of building a log filename from it.

if ! submission=$(sbatch --parsable smoke.sbatch); then
  echo "Submission failed; check the site policy and error above. No job ID was captured." >&2
  exit 1
fi
if [[ "$submission" =~ ^([0-9]+)(;([^;[:space:]]+))?$ ]]; then
  JOB_ID=${BASH_REMATCH[1]}
  CLUSTER=${BASH_REMATCH[3]:-}
else
  printf 'Unexpected sbatch --parsable response: %s\n' "$submission" >&2
  exit 1
fi
printf 'Slurm accepted job %s' "$JOB_ID"
if [[ -n "$CLUSTER" ]]; then printf ' on cluster %s' "$CLUSTER"; fi
printf '\n'

An assigned ID means the controller accepted the submission, not that the job started or succeeded. A pending state means it is waiting; it is not a failed run. Use the same captured ID to check the queue and accounting. A federated response includes a cluster suffix, which the commands below pass to Slurm's -M option.

if [[ -n "$CLUSTER" ]]; then
  squeue -M "$CLUSTER" -j "$JOB_ID"
  if ! accounting=$(sacct -M "$CLUSTER" --noheader --parsable2 --jobs="$JOB_ID" --format=JobID,State,ExitCode); then
    echo "Accounting query unavailable; check site documentation and try later." >&2
    exit 1
  fi
else
  squeue -j "$JOB_ID"
  if ! accounting=$(sacct --noheader --parsable2 --jobs="$JOB_ID" --format=JobID,State,ExitCode); then
    echo "Accounting query unavailable; check site documentation and try later." >&2
    exit 1
  fi
fi
printf 'Accounting rows (job and possibly step records):\n%s\n' "$accounting"
allocation=$(printf '%s\n' "$accounting" | awk -F'|' -v id="$JOB_ID" '$1 == id { print $2 "|" $3; exit }')
if [[ -z "$allocation" ]]; then
  echo "No allocation accounting row yet. This is not evidence of failure; wait or ask the site about accounting." >&2
  exit 1
fi
if [[ "$allocation" != "COMPLETED|0:0" ]]; then
  printf 'Success is not confirmed (allocation record: %s). Pending or unavailable records do not prove failure.\n' "$allocation" >&2
  echo "Do not resubmit automatically. Check the state and site guidance first." >&2
  exit 1
fi
if grep -Fq 'Sum 1..100: 5050' "slurm-${JOB_ID}.out"; then
  echo "Confirmed COMPLETED/0:0 and expected synthetic output."
else
  echo "Job reports COMPLETED/0:0 but the expected output line is absent; inspect the job output and working directory." >&2
  exit 1
fi

Use the allocation row whose JobID exactly matches $JOB_ID, not a step row such as .batch. The success check above requires both COMPLETED and exit code 0:0 before checking the expected output in slurm-$JOB_ID.out. A zero exit code alone does not validate a scientific workflow.

Worked reading: if the row for your captured ID reads ID|PENDING|0:0, the state is still pending, so the success check correctly refuses to inspect success output. Wait and check again according to site guidance. If a terminal row instead reads ID|FAILED|1:0, the job failed with a non-zero exit status; after recording that final state, inspect slurm-$JOB_ID.err and the job's output to diagnose it. These rows are illustrative, not results from a real job.

If the job is pending, it may not have created logs yet. If the queue no longer shows it or sacct has no row, accounting may be delayed or unavailable. Neither case means failure: wait, check the site status guidance, or contact support. Do not resubmit in a loop. First inspect the recorded state. For a terminal failure, use the same validated ID to inspect its error log, if the site stores logs in this shared working directory:

cat "slurm-${JOB_ID}.err"
cat "slurm-${JOB_ID}.out"

For a module error, confirm that the documented module system is initialized inside the batch shell and that the exact module is available there. For a submission rejection, inspect the site-required partition, account, resource limits, and directive syntax. For an out-of-memory or time-limit state, consult accounting and site policy before changing resource requests. If accounting is disabled or delayed, use the site's documented status route rather than inferring failure from absent records or files.

What this lesson did and did not test

The R script and arithmetic are synthetic and runnable locally; the optional Python route uses the same arithmetic. Local execution does not test Slurm. The batch-script syntax and failure/success paths are tested with local command mocks only; mocks do not schedule a job, load a real site module, exercise a compute node, or verify a cluster's accounting and filesystem behavior. No real cluster, scheduler, paid service, private research material, account, or email is used by these local checks. Site setup, permissions, resource policy, and actual job execution remain for your administrator-approved environment.

After this reference, continue to Writing Slurm Job Scripts to learn how to adapt directives and script structure for a real workload. If you do not have HPC access or need a different prerequisite, return to Start Here; HPC is optional.

References

• Slurm sinfo manual, partition and node reporting.
• Slurm squeue manual, output formats and job state codes.
• Slurm sbatch manual, batch submission, directives, and output files.
• Slurm sacct manual, job accounting.
• Environment Modules module manual, including shell initialization.
• Lmod documentation. The installed module system and commands may differ by site.