HPC Troubleshooting and Support

Use the job record and its logs to narrow down a failure before changing the submission script. Slurm reports what happened to the job; the application log and your script help explain why.

Learning objectives and prerequisites

By the end of this lesson you should be able to:

• Use core Slurm tools (sacct, scontrol, squeue, tail) to collect job evidence.
• Use common job states and exit-code evidence to narrow possible resource, software, and environment explanations without treating one clue as a confirmed cause.
• Prepare a compact, non-sensitive evidence summary for a support request.
• Choose an evidence-based next check, distinguish a plausible cause from a confirmed cause, and know when a change or support escalation requires local guidance.

Prerequisites: you have access to the cluster where your jobs ran, the original submission script or command, and the job ID(s) you want to investigate. Familiarity with basic shell commands, your cluster's module or environment system (Modules or Conda), and the bioinformatics tool(s) used in the job is assumed.

Start with evidence, then choose a fix

A failed job can reflect an application error, a missing input, an environment mismatch, a scheduler limit, or a cluster-side event. Capture the job state and relevant log lines before resubmitting. Slurm commands and accounting availability vary by site, so check local cluster guidance when a field or command is unavailable.

Reproducible workflow: collect evidence and try local fixes

Work through the records first, then inspect the script and environment. Make one change at a time and rerun only a small, representative test when software and data policy allow it.

Files you should have before starting

• Your submission script (example: run_alignment.sbatch)
• The job ID (example: 1234567) from sbatch output or the cluster web UI
• Any generated log files (example: slurm-1234567.out and optional slurm-1234567.err)

1) Collect Slurm accounting and state information

Purpose: inspect the recorded job state, script exit status, elapsed time, memory request, and reported peak resident memory. Replace 1234567 with your job ID. Accounting availability, retention, and populated fields depend on cluster configuration.

# Show job summary: JobID, name, state, exit code, elapsed walltime, requested memory, and peak memory usage
sacct -j 1234567 --format=JobID,JobName,State,ExitCode,Elapsed,ReqMem,MaxRSS

# If your cluster supports it, get a single-line overview using seff
seff 1234567

# Show detailed scontrol job info (node allocations, submit host, environment variables)
scontrol show job 1234567

What these produce:

• sacct: one or more records for the allocation and its steps (for example, 1234567.batch). Compare the relevant step as well as the job record; reported measurements depend on site accounting configuration.
• seff, when installed: a site-provided summary of resource use. Treat it as a convenience, not a substitute for site documentation.
• scontrol show job: job details while retained and available. Historical records may no longer be available there; use sacct for accounting history where configured.

For a job that is still pending or running, inspect the live queue with squeue -j 1234567 (replace the example ID). Its state and, for a pending job, the displayed reason can guide the next check; queue visibility is transient and does not replace sacct for retained accounting history. Output and access can vary by site.

2) Inspect the job log(s)

Purpose: locate application errors and shell tracebacks. Slurm commonly combines output in slurm-<jobid>.out unless the script sets --output or --error. Confirm paths in the batch script; separate files are not guaranteed.

# Show the last 200 lines of the default log file (adjust path as needed)
tail -n 200 slurm-1234567.out

# If stderr was split, check both
tail -n 200 slurm-1234567.err
tail -n 200 slurm-1234567.out

What to look for:

• Stack traces, Python/R exceptions, or explicit error messages from the tool (e.g., "FileNotFoundError" or "Segmentation fault").
• Messages from the job scheduler injected into the file (e.g., "Killed" or "Memory killed").
• Missing module/command messages like "command not found" or "ModuleNotFoundError".

3) Confirm the execution environment

Purpose: verify that the same module or Conda environment will be used in a resubmission. A common error is forgetting to load modules or activate Conda in the batch script.

# If your cluster uses Modules:
module avail          # lists available modules
module list           # lists modules currently loaded in an interactive session

# If using Conda from within the batch script, ensure the activation step is present
# Example lines that should appear in your submission script (run_alignment.sbatch):
# source /path/to/miniconda3/etc/profile.d/conda.sh
# conda activate myenv

# In an interactive node (if permitted), check tool versions
command -v bwa
bwa 2>&1 | head -n 1  # consult your cluster/tool documentation for version syntax

No files are produced; these are checks you perform interactively. If a tool is missing or the wrong version is found, adjust the batch script to load the correct module or activate the correct environment.

4) Minimal re-run for debugging (short, small resource test)

Purpose: avoid re-running a full job. This generic BWA illustration expects a single-end gzip-compressed FASTQ, a BWA-indexed reference, and permission to use a small subset of the data. Obtain inputs only from an approved cluster location; do not copy protected data to an unapproved workstation. Run from a scratch/work directory with enough space. First set real paths and stop if any check fails:

# Replace these examples with paths approved for your cluster and project.
READS="/approved/path/reads.fastq.gz"
REF="/approved/path/ref/genome.fa"
test -r "$READS" && test -r "$REF" || { echo "STOP: input or reference is not readable"; exit 1; }
gzip -t "$READS" || { echo "STOP: compressed input is damaged or not gzip"; exit 1; }
for suffix in amb ann bwt pac sa; do
  test -r "${REF}.${suffix}" || { echo "STOP: missing BWA index ${REF}.${suffix}"; exit 1; }
done

# A FASTQ record has four lines. Take 2500 records (10,000 lines), preserving record boundaries.
zcat "$READS" | head -n 10000 | gzip -c > test_reads.fastq.gz
gzip -t test_reads.fastq.gz || { echo "STOP: subset gzip check failed"; exit 1; }
lines=$(zcat test_reads.fastq.gz | wc -l)
test "$lines" -gt 0 && test "$((lines % 4))" -eq 0 || { echo "STOP: empty or incomplete four-line FASTQ records"; exit 1; }
printf 'Subset contains %s complete FASTQ records\n' "$((lines / 4))"

# Run only on an allowed interactive/compute node, using site-approved resources.
# Single-end example. Keep output in scratch; check the BWA and reference conventions locally.
bwa mem -t 4 "$REF" test_reads.fastq.gz > test_reads.sam

# Or submit a tiny batch job to test environment and basic resource behavior
cat > tiny_test.sbatch <<'SBATCH'
#!/bin/bash
#SBATCH --job-name=tinytest
#SBATCH --time=00:10:00
#SBATCH --cpus-per-task=4
#SBATCH --mem=4G
#SBATCH --output=tinytest-%j.out

module load bwa
# This site-specific module name and the paths below must be checked before submission.
REF="/approved/path/ref/genome.fa"
test -r "$REF" || { echo "STOP: set REF to a readable reference"; exit 1; }
bwa mem -t 4 "$REF" test_reads.fastq.gz > tinytest.sam
SBATCH

sbatch tiny_test.sbatch

For paired-end reads, create R1 and R2 subsets from the same source records using a paired-aware method, then pass both files to BWA in mate order; never downsample mates independently. Adapt the command, reference/index format and resource request to the original pipeline and site policy. Files produced: test_reads.fastq.gz, test_reads.sam (or tinytest-%j.out and tinytest.sam). The FASTQ line-count check confirms only a non-empty, four-line record boundary, not that sequence, quality and header fields are biologically valid or mates match. A successful small test shows only that this particular subset worked in that environment; it does not establish that the full input fits requested resources or rule out data-specific failures. Stop and ask cluster support if paths, permitted compute location, index files, or input format cannot be verified.

Output inspection and interpretation

After you collect the outputs above, here are the common pieces of evidence and how to interpret them, including what a result does not prove.

sacct and seff output

• OUT_OF_MEMORY is direct scheduler evidence that Slurm recorded an out-of-memory event. A large MaxRSS relative to ReqMem can support a memory-limit hypothesis, but values may be missing or reported per step and do not alone prove which process exceeded a limit.
• COMPLETED with ExitCode 0:0 means the recorded job script exited successfully. If expected files are absent, check output redirection, relative paths, conditional steps, and working directory; successful status does not verify scientific correctness or output completeness.
• An exit code of 137 is often produced when a process is terminated by signal 9 in shell conventions, but it does not identify who sent the signal or why. Do not diagnose an out-of-memory kill from 137 alone. Check Slurm state, step records, site logs, and application output.

Log file contents

• Tool-specific error messages (e.g., "FileNotFoundError: data/reads.fastq"): this implicates files or paths; check SLURM_SUBMIT_DIR and that all input files existed at submission time. This proves a missing-file condition if the path is absolute or the file's absence can be verified.
• "command not found" points to command resolution in that execution context. Check module and environment setup, executable paths, and whether the command exists on compute nodes. The message alone does not establish why it was absent.
• Segmentation faults or "Bus error": could be corrupted input, a software bug, or exceeding memory/stack limits. These messages require more investigation (e.g., trying a small test or a different software version).

Troubleshooting: symptoms, possible causes, and next checks

The examples below suggest evidence to examine, not a guaranteed diagnosis or universal fix. Confirm resource, software, and environment changes against job records and local policy.

Symptom: Exit code 127 or "command not found"

Possible cause: the executable was unavailable through the batch job's PATH or environment. Check module setup, activation, and command spelling.

Fixes:

• Add explicit module load lines to the batch script: module load bwa/0.7.17 (replace with your cluster’s module name).
• For Conda, ensure the activation lines are present and use the absolute path to conda's initialization script:

  source /path/to/miniconda3/etc/profile.d/conda.sh
  conda activate myenv

• Test the script in an interactive job or short test job to confirm the tool is available.

Symptom: Exit code 137

Possible cause: a memory limit or another source sent the process a kill signal. Exit code 137 alone cannot distinguish these causes. Look for OUT_OF_MEMORY or site evidence before changing memory requests.

Fixes:

• If Slurm or site evidence supports a memory limit, adjust the request according to cluster rules (for example, --mem or --mem-per-cpu). Use observed usage and a reasonable margin; a larger request is not proof of the cause.
• Where supported, test whether fewer concurrent workers or smaller input batches reduce peak memory. Record the exact input subset and parameters.
• Profile the program memory usage with test inputs or tools like /usr/bin/time -v to measure peak RSS locally before resubmitting.

  /usr/bin/time -v my_program args > out.txt 2> time-stats.txt

Symptom: TIMEOUT or job ended with State=TIMEOUT

Likely cause: requested walltime (--time) was too small.

Fixes:

• Increase the walltime in the batch script, but be realistic: over-requesting may increase queue wait time. Use seff from previous successful runs to estimate realistic walltime.
• Enable periodic checkpointing or implement pipeline restart points if the tool supports it.

Symptom: Job disappears from queue or is CANCELLED by admin

Likely cause: administrative intervention (e.g., maintenance) or cluster policy (preemption/priority).

Fixes:

• Check cluster notices and the recorded state. CANCELLED can reflect user or administrator action; PREEMPTED indicates preemption. Contact support when the reason is unclear.
• If preemption is likely, consider using a different partition/quality-of-service (QOS) with fewer preemptions for long jobs.

Symptom: Application crashes with Segmentation fault

Likely causes: corrupted input, incompatible binary, or insufficient memory.

Fixes:

• Validate inputs (e.g., check FASTQ/FASTA integrity with md5sum or format validators).
• Check whether the executable and linked libraries are compatible with compute nodes. Test another version only when the cluster provides it and record that version; compiling from source is not a first diagnostic step.
• Test a small dataset to reproduce the crash quickly.

How to prepare an effective support request

If you need to contact cluster support, include a concise, privacy-respecting set of items. Avoid attaching raw data that may be sensitive. A useful support-ready report includes:

• Job ID(s) and approximate timestamps (submit and end times).
• Exact sbatch command or submission script (paste the file body or attach a text file).
• The specific command that failed inside the script (not only the script name).
• Relevant sacct output and, if still available, scontrol show job output. Include the command and fields; redact account names, paths, hostnames, environment values, or other identifiers according to local policy. Avoid copying full environment dumps.
• The smallest useful log excerpt, including the first relevant error and surrounding context. Redact sensitive paths, sample identifiers, credentials, and data; state that redactions were made.
• Steps you have already tried, with the request, test input, and observed result; do not report a test or change that you did not perform.

Example template to paste into a ticket:

Job ID: 1234567
Submit host: (redact or omit if sensitive)
Partition: (if relevant and permitted)
Submission script: (attach a redacted copy)
Command that failed: (exact command, with sensitive paths and sample identifiers redacted)

sacct output:
( paste sacct -j 1234567 --format=JobID,State,ExitCode,ReqMem,MaxRSS here )

Log excerpt (last 100 lines):
( paste relevant log lines here )

Steps attempted:
- Checks already performed: (state what you actually checked)
- Small test: (input subset, command, and observed result, if performed)
- Resource changes: (request and recorded result, if changed)

Limitations, version and data caveats, and responsible use

• Slurm version, accounting plugins, retention, and site policy affect commands and fields. Check cluster documentation; this lesson does not certify commands against a particular installation.
• seff may not be installed. /usr/bin/time -v measures a process launched with that command; it does not replace scheduler accounting for a distributed job.
• Always avoid including raw sequence data or personally identifiable information (PII) in support tickets. If support needs such data, coordinate a secure transfer channel with the administrators.
• Administrative fixes (node reboots, system logs) may be restricted to support staff; do not ask support to run destructive debugging commands (e.g., rm -rf). If a support request could reveal sensitive information, ask about redaction procedures first.
• Performance on a local workstation may not reflect cluster behavior (different CPU architecture, memory bandwidth, or MPI libraries). Use a short test on the cluster itself where possible.

Next-step checklist (concise)

• Collect available sacct records and the relevant log excerpt. Use scontrol show job only if details remain available.
• If policy and software permit, try a small, representative test; record its limits and avoid treating success as proof about the full run.
• Check and explicitly set module or Conda activation lines in your sbatch script.
• If resource evidence supports a limit, adjust one relevant request or concurrency setting using observed use and site guidance; record the change and outcome.
• If unresolved, submit a concise support ticket including the template shown above and note any privacy redactions.

References

• Slurm official documentation (SchedMD)
• sacct: Slurm accounting tool
• squeue: view jobs in the scheduling queue
• scontrol: display and manage Slurm state
• Slurm job state codes
• SchedMD Slurm troubleshooting guide, covering unscheduled jobs and other Slurm failures.