Reproducible Project Structure

A reproducible analysis needs a clear boundary between inputs, code, derived files and reported results. This walkthrough builds that structure and records enough provenance to investigate a run later. R is the main analysis language, with R Markdown reports kept alongside the analysis code. The layout is a starting point; it does not supply a QC analysis or prove that an analysis is scientifically valid.

Learning objectives and prerequisites

By the end, you can create a role-oriented project tree, record checksums and software versions, and prepare a run wrapper that logs its code revision and configuration path. You will also know what these records can and cannot establish.

Prerequisites: basic Bash and Git; a Unix-like shell with mkdir, sha256sum, find, and git. The main wrapper expects Rscript. Conda, renv, and the optional Python inventory require their respective tools. Run the commands from the project root unless a step says otherwise.

1. Create the project skeleton

From the parent directory, create a project directory. Replace project with your own name. The brace expansion below is Bash syntax.

mkdir -p project/{config,data/{raw,processed},envs,scripts,results/{figures,tables},logs}
touch project/README.md project/config/.keep project/data/raw/.keep
cat > project/.gitignore <<'EOF'
# Keep raw data out of Git; store it in an approved, backed-up location.
data/raw/
data/processed/
results/
*.pyc
__pycache__/
.env
EOF

Use data/raw/ for source inputs, data/processed/ for regenerable intermediates, scripts/ for code, results/ for selected outputs, logs/ for run records, config/ for parameters and provenance, and envs/ for dependency manifests. Treat raw inputs as immutable by project policy, and retain a separate backup. A .gitignore rule prevents ordinary unforced Git additions; it is not access control or a backup.

2. Keep analysis settings explicit

This YAML fragment illustrates the kinds of values a particular analysis might configure. It is not a complete configuration contract: the example QC program mentioned below does not exist in this scaffold, and no code here reads or validates these keys. Define the schema and units in the README and implement matching parsing and validation in your analysis.

# config/analysis.yml (illustrative only)
threads: 4
reference: GRCh38
min_counts: 10
input_counts: data/raw/counts.tsv
output_dir: data/processed/qc

For a real run, document whether paths are relative to the project root, what file format and delimiter are accepted, which columns are required, and what each parameter means. A reference label alone does not identify an exact reference bundle: record its source, release/accession, checksum, and any index used.

3. Record input and software provenance

From the project root, generate one checksum record per file directly inside data/raw/. The output stores relative names, making verification from the same root straightforward.

find data/raw -maxdepth 1 -type f -print0 | sort -z | xargs -0 -r sha256sum > config/raw_checksums.sha256
sha256sum -c config/raw_checksums.sha256

The second command reports whether each recorded file matches its digest and returns failure if a check fails. This covers only files directly in data/raw/, not nested directories. Rebuild the manifest only after an intentional, documented input change; do not overwrite the evidence of an unexplained mismatch. Checksum agreement detects byte changes, not source authenticity or scientific suitability.

From the project root, record versions of the tools available in the environment. R and Git are used in the main example. Python is recorded only when you use an optional Python tool. Afterward, inspect config/versions.txt to confirm which entries were captured.

Rscript --version > config/versions.txt 2>&1
Rscript --vanilla -e 'sessionInfo()' >> config/versions.txt 2>&1
git --version >> config/versions.txt 2>&1
# Optional, only if this project uses a Conda environment:
# conda env export --no-builds > envs/environment.yml
# Optional Python distributions, only when a Python tool is used:
# python -m pip freeze > envs/requirements.txt

For an R project, initialize renv deliberately from its project root, review the packages found, and keep the lockfile with the code. These R-console commands require the renv package; initialization changes the project environment, so use a fresh teaching project first.

# Run in the R console from the project root after installing renv.
renv::init(bare = TRUE)
renv::snapshot(lockfile = "renv.lock")

Keep report sources such as reports/analysis.Rmd under version control. Write rendered reports and derived figures under results/; record the command used to render them. A lockfile records selected packages, not the source data, system libraries, or scientific validity.

--no-builds removes Conda package build specifications; it does not guarantee cross-platform recreation. pip freeze records installed distributions but does not itself capture Python, operating-system, or system-library details. Choose the manifest that matches the environment manager actually used, and record the platform separately when relevant.

4. Add a run wrapper

The following wrapper records the project path, current Git revision when available, configuration path, and command output. It assumes a project with scripts/, config/, and logs/. Save as scripts/run_qc.sh; adapt the final command to a real analysis implementation.

#!/usr/bin/env bash
set -euo pipefail

PROJECT_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
CONFIG="$PROJECT_ROOT/config/analysis.yml"
mkdir -p "$PROJECT_ROOT/logs"
LOG="$PROJECT_ROOT/logs/run_$(date +%Y%m%dT%H%M%S).log"
COMMIT="$(git -C "$PROJECT_ROOT" rev-parse --short HEAD 2>/dev/null || echo 'no-git')"

{
  echo "Run started: $(date --iso-8601=seconds)"
  echo "Project: $PROJECT_ROOT"
  echo "Git commit: $COMMIT"
  echo "Config used: $CONFIG"
  echo "Config SHA-256: $(sha256sum "$CONFIG")"
  echo "Config snapshot:"
  cat "$CONFIG"
  echo "---"
  # Replace this placeholder with the analysis you have implemented.
  Rscript --vanilla "$PROJECT_ROOT/scripts/qc_counts.R" --config "$CONFIG"
  echo "Run finished: $(date --iso-8601=seconds)"
} >& "$LOG"

The root calculation uses scripts/.., because the script resides one directory below the project root. The braces send both standard output and standard error to the log. The wrapper records the configuration path, its SHA-256 digest, and a copy of its contents in that run's log: the path alone would not identify parameter values if the file later changed. The digest helps detect a changed copy when compared with a retained file; the logged snapshot records the values used. With set -e, a failing analysis command normally stops the script before the finish line; Bash has documented exceptions to -e, so use explicit status handling if the wrapper grows more complex. The example uses GNU/Linux date --iso-8601; other systems may need a portable timestamp command.

qc_counts.R is an expected user-supplied R implementation, not included here. It must accept the shown --config argument, parse the documented configuration, read the specified count table, and write the documented output. Until that program and its input contract exist, the wrapper is a template and will fail at the analysis step. Do not treat a created log or output filename as evidence of valid metrics.

5. Initialize version control deliberately

Review your ignore rules and ensure no restricted data, credentials, or large generated files are staged. The following commands change Git metadata and create a commit in the project; inspect the staged file list before committing.

cd project
git init
git add README.md config scripts .gitignore
git status --short
git commit -m "Initial project structure and documentation"
git rev-parse --short HEAD > config/git_commit.txt

The last command records the commit identifier in a file after the commit, so that file itself is not included in that commit. The wrapper reads the current Git revision at run time; keep code and configuration changes committed before running when you need a stable revision identifier. Avoid tracking private data even if a broad directory is ignored.

6. Check the run and diagnose failures

• Checksum check fails: stop and compare the named file against its source or backup. Determine whether it changed intentionally before regenerating the manifest.
• R reports a missing package: check Rscript --version and .libPaths() in the same environment as the wrapper. Inspect the project lockfile and restore the intended project library deliberately; do not install an arbitrary version just to bypass the error.
• Configuration or input cannot be found: inspect the recorded absolute project/config paths, then verify that configured input paths resolve from the project root. Check delimiter, header, and required columns against the parser contract.
• Metrics look implausible: inspect a small input excerpt and the analysis code's parsing and filtering steps. A structural check such as non-empty rows is not a universal biological QC threshold.
• No Git revision appears: confirm that the project root is a Git repository and that it has at least one commit. A missing revision is a provenance gap, not proof the computation failed.

A log can help establish which command ran and what it printed. Matching checksums and recorded versions improve traceability. None establishes that the software is correct, assumptions are appropriate, or biological interpretations are valid. Review the methods and outputs, and use suitable independent validation where the scientific question requires it.

Further project conventions

Use filenames that communicate role and, where useful, sortable dates, for example 2026-08-15_sample01_R1.fastq.gz. Adopt a consistent convention for the project rather than treating one naming rule as universal. Keep sensitive or personally identifiable information out of version control; document access procedures and applicable approvals separately. For reference sequences, record exact release/accession and checksum, plus the index files used. Containers can complement environment manifests, but do not by themselves capture input data, external services, hardware behavior, or every source of variation.

References

• Git documentation
• Conda environment export documentation
• renv documentation
• Sandve et al., Ten simple rules for reproducible computational research