Reference Genomes and Annotation Databases

A reference is a matched set: the sequence assembly defines the coordinates, and an annotation release places genes and other features on that sequence. This walkthrough helps you select a compatible pair, check what you downloaded, and leave a manifest that records exactly what the analysis used.

What you will produce

At the end, your project will contain a genome FASTA, a gene annotation file, a FASTA index, two sorted contig-name lists, and a manifest with source URLs and local SHA-256 checksums. The examples use human GRCh38.p14 and GENCODE release 46. They are examples, not a universal recommendation: choose the assembly and annotation appropriate to your organism and analysis.

You need a POSIX shell, curl, sha256sum, gzip, standard text tools (awk, sort, comm), and samtools for indexing. The commands below have not been run against the full human reference files in this review. Large reference downloads require substantial disk space and time.

1. Choose a matched assembly and annotation

Record both the assembly identity and the annotation release. GRCh38.p14 is identified by RefSeq assembly accession GCF_000001405.40; GENCODE labels release 46 as GRCh38.p14. Those labels make this a plausible compatible example pair, but you must still decide which sequence-region scope fits your pipeline.

GENCODE provides distinct annotation sets for reference chromosomes only (CHR), the primary assembly (PRI, chromosomes and scaffolds), and all regions (ALL, including patches and alternate loci). The example uses the ALL GTF and its matching GENCODE GRCh38.p14 genome FASTA, so both downloads describe the same broad sequence-region scope. If your pipeline requires primary-assembly-only data, select the corresponding primary-assembly GTF and FASTA instead. Do not pair a chromosome-only GTF with an all-regions FASTA and then treat expected unannotated contigs as a failure.

2. Download and record the exact files

Run these commands from the project directory. They create refs/, save the official GENCODE release-46 genome FASTA and comprehensive all-regions GTF, and fail on HTTP errors instead of silently accepting an error page as a data file.

mkdir -p refs
cd refs

FASTA_URL="https://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_46/GRCh38.p14.genome.fa.gz"
GTF_URL="https://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_46/gencode.v46.chr_patch_hapl_scaff.annotation.gtf.gz"

curl --fail --location --output GRCh38.p14.genome.fa.gz "$FASTA_URL"
curl --fail --location --output gencode.v46.chr_patch_hapl_scaff.annotation.gtf.gz "$GTF_URL"

# These are locally computed SHA-256 values, not provider-issued checksums.
sha256sum GRCh38.p14.genome.fa.gz gencode.v46.chr_patch_hapl_scaff.annotation.gtf.gz

Keep the command output with your run records. The release page identifies the exact filenames, assembly, and region scope; SHA-256 lets you later check whether your local copy has changed. This example does not download a provider checksum file. If the provider publishes a checksum for the chosen artifact, verify it using the provider's documented format as well.

3. Index the FASTA and inspect its contents

samtools faidx creates an index used for random access. It accepts plain FASTA or BGZF-compressed FASTA; a file compressed with ordinary gzip is not necessarily BGZF. GENCODE’s downloadable .fa.gz is a gzip download, so decompress a working copy before indexing. Preserve the original compressed file and do not overwrite it.

gzip --test GRCh38.p14.genome.fa.gz
gzip --keep --decompress GRCh38.p14.genome.fa.gz
samtools faidx GRCh38.p14.genome.fa

cut -f1 GRCh38.p14.genome.fa.fai | sort -u > fasta_contigs.txt
zcat gencode.v46.chr_patch_hapl_scaff.annotation.gtf.gz \
  | awk 'BEGIN { FS="\t" } !/^#/ { print $1 }' \
  | sort -u > gtf_contigs.txt

The FASTA index lists sequence names and lengths. The GTF command emits distinct sequence names from feature rows, skipping comment lines. Empty output, a gzip integrity error, a failed index, or a name that looks like an HTML response calls for inspecting the downloaded file and command error before continuing.

4. Compare contig names without forcing false equality

echo "In FASTA but not named in GTF:"
comm -23 fasta_contigs.txt gtf_contigs.txt | head

echo "In GTF but not named in FASTA:"
comm -13 fasta_contigs.txt gtf_contigs.txt | head

comm expects sorted inputs; the preceding commands sort both lists. A name in the GTF but not in the FASTA is a direct compatibility warning. FASTA-only names can be expected when some assembly regions have no annotated features, depending on the chosen GENCODE scope. Review the complete differences and assembly report before deciding. Do not conclude that the files match just because the first few lines printed by head are empty, and do not rename contigs by a blind global substitution.

When a well-understood naming difference is the only mismatch, build an explicit mapping table and apply a documented conversion to a copy of the annotation. Confirm that the mapping is one-to-one for the sequences you need, retain the original file, and rerun the comparison. A rule that maps 1 to chr1 is not enough for every chromosome, mitochondrial sequence, patch, or alternate contig.

5. Inspect annotation identifiers and coordinates

GTF rows have nine tab-separated fields. The first field names the sequence, fields four and five give the start and end, and field nine contains attributes such as gene and transcript identifiers. Inspect a few records before using them in joins or interval operations:

zcat gencode.v46.chr_patch_hapl_scaff.annotation.gtf.gz \
  | awk 'BEGIN { FS="	" } !/^#/ { print; n++; if (n == 3) exit }'

GENCODE records use Ensembl-style identifiers, often with version suffixes. Keep the original IDs and the release context when mapping to gene symbols for figures; symbols are useful labels but can change or be reused. Record the mapping file if you create one.

Coordinate conventions differ by format. GTF/GFF feature starts and ends are conventionally one-based and inclusive; BED uses a zero-based start and an end that is excluded. Converting between them requires changing the start coordinate, not merely relabeling a file. Check the specific downstream tool’s documentation before conversion, and never infer a coordinate convention from a filename alone.

6. Write a manifest and carry it with the analysis

The earlier steps leave the shell inside refs/. The first line below returns to the project directory before creating refs/reference_manifest.tsv. If you have already returned to the project root in a fresh terminal, omit that first cd ... The block uses the compressed-file checksums and writes the header plus both tab-separated rows with the exact source URLs and file paths:

cd ..
FASTA_SHA=$(sha256sum refs/GRCh38.p14.genome.fa.gz | awk '{print $1}')
GTF_SHA=$(sha256sum refs/gencode.v46.chr_patch_hapl_scaff.annotation.gtf.gz | awk '{print $1}')
printf 'provider\tresource\tassembly\trelease\turl\tsha256\tfile_path\tregion_scope\nGENCODE\tgenome FASTA\tGRCh38.p14\t46\thttps://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_46/GRCh38.p14.genome.fa.gz\t%s\trefs/GRCh38.p14.genome.fa.gz\tALL\nGENCODE\tannotation GTF\tGRCh38.p14\t46\thttps://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_46/gencode.v46.chr_patch_hapl_scaff.annotation.gtf.gz\t%s\trefs/gencode.v46.chr_patch_hapl_scaff.annotation.gtf.gz\tALL\n' \
  "$FASTA_SHA" "$GTF_SHA" > refs/reference_manifest.tsv

# Confirm a header plus two rows, and reject empty or placeholder digests.
test "$(wc -l < refs/reference_manifest.tsv)" -eq 3 && \
  awk -F '\t' 'NR > 1 && (NF != 8 || length($6) != 64 || $6 !~ /^[0-9a-f]+$/) { exit 1 }' refs/reference_manifest.tsv && \
  echo "Manifest structure and SHA-256 fields look complete"

The checks confirm three lines, eight tab-separated fields per data row, and 64-character lowercase hexadecimal digests; they do not validate the reference choice. Include the decompressed FASTA and index in your project’s storage policy as appropriate, or record how to recreate them from the preserved compressed source. Add download date, tool versions, any conversion mapping, and the reference used to build an aligner index when those details matter to your workflow.

Diagnose common mismatches

Before using the reference

• Confirm the species, assembly accession/version, annotation release, and region scope fit the analysis.
• Keep the original FASTA/GTF and preserve the hashes, exact URLs, and manifest alongside analysis code or data.
• Check contig names and lengths, annotation format, identifier conventions, and the coordinate expectations of each downstream tool.
• Document conversions and derived indexes. A clean software check does not establish that the selected reference is appropriate for every biological question.

References

• GENCODE human release 46: GRCh38.p14 files and region scopes
• NCBI Datasets: GRCh38.p14 assembly GCF_000001405.40
• samtools faidx manual
• UCSC data file formats: BED and GFF
• NCBI Genomes FTP file documentation