Evaluating External scGPT & Geneformer Embeddings

Use base R to validate embedding and metadata CSVs, align rows by cell ID, reject donor leakage, and evaluate donor-held-out cell-label transfer with squared-Euclidean k-nearest neighbors. The same question can be evaluated with the optional existing Python workflow. Model-specific embedding generation remains an upstream, separately documented step.

Goal and assumptions

• Check the exact CSV schema, unique cell identifiers, one-to-one identifier coverage, numeric finite coordinates, and metadata labels.
• Keep all cells from each donor in one split and reject overlap between the reference and held-out test sets.
• Run deterministic squared-Euclidean kNN; report accuracy, macro-F1, per-class F1, donor accuracy, neighbor IDs, distances, and top-vote fraction.

Upstream embedding handoff (not executed here)

Real model vectors must be generated separately with the exact official implementation, checkpoint, tokenizer/vocabulary, organism, preprocessing, embedding layer, and pooling documented for that model. Keep a record of those choices and the export provenance. Do not label toy vectors as scGPT or Geneformer output.

• scGPT: use the official reference-mapping tutorial or integration tutorial. Check the chosen checkpoint and paired vocabulary and follow its actual preprocessing instructions; this lesson assumes no generic model API.
• Geneformer: follow its official getting-started documentation and tokenization and embedding examples. Tokenization, gene identifiers, species, model version, and checkpoint differ from scGPT.

No pretrained output, model weights, inference code, GPU requirement, or heavyweight model installation is included or tested here. Donor holdout in this evaluator does not establish that a checkpoint never saw those donors or their study during pretraining.

Input contract

Use two UTF-8 comma-separated files. The embeddings file has a cell_id column followed by one or more embedding_* numeric columns. Metadata has exactly cell_id,donor_id,label,split, with split equal to train or test. Each identifier appears exactly once in each file; row order may differ because the evaluator joins by ID. Donors must be exclusive to one split.

cell_id,embedding_0,embedding_1
ref_T,0,0
ref_B,10,10
query_T,0.1,0.2
query_B,9.8,10.1

cell_id,donor_id,label,split
query_B,heldout_2,B-cell,test
ref_B,reference_2,B-cell,train
ref_T,reference_1,T-cell,train
query_T,heldout_1,T-cell,test

These example rows are a deliberately easy constructed fixture. Their separation and score are software-check facts only. Choose the split unit, checkpoint, preprocessing, distance metric, and k using reference-only validation, never held-out labels.

Run the evaluator: R main route, Python optional

Download evaluate_embeddings.R, the synthetic embedding fixture, and the matching metadata fixture into one new working folder. The R script needs base R only. The adjacent Python command is a complete optional implementation of the same held-out kNN task; download foundation_embedding_workflow.py into that folder to use it.

Rscript evaluate_embeddings.R synthetic_embeddings.csv synthetic_metadata.csv demo 1

python3 foundation_embedding_workflow.py --embeddings synthetic_embeddings.csv --metadata synthetic_metadata.csv --output demo.json -k 1

The R route writes demo_predictions.csv and demo_metrics.csv; the Python route writes a JSON report with the same core accuracy, macro-F1, per-class F1, and donor-level accuracy plus a confusion matrix and detailed per-cell neighbor information. Both runs use identical synthetic rows and are deliberately easy software checks: the two query cells map to constructed labels, with accuracy and macro-F1 of 1.0 only for this toy fixture. Neither computes model embeddings or supports biological claims.

Both evaluators reject donor leakage, mismatched IDs, invalid numeric values, and oversized inputs. The R script refuses CSVs above 10 MiB and comparison workloads above 10 million reference-cell × test-cell × dimension terms; it does not subsample. The Python workflow applies the same teaching limits. Use a fresh result path. The Python test file constructs its own small fixtures; download test_foundation_embedding_workflow.py beside the Python utility and run python3 -m unittest -v test_foundation_embedding_workflow.py.

For the optional report, download foundation_embedding_evaluation.Rmd beside the R script and fixtures. Render with Rscript -e 'rmarkdown::render("foundation_embedding_evaluation.Rmd")' from that directory; rendering requires the rmarkdown package and Pandoc, but the evaluator itself does not.

Interpret results and limitations

Inspect support and misclassified labels, per-class and donor-level metrics, and nearest neighbors before drawing conclusions. Performance may reflect tissue, site, batch, class balance, inconsistent labels, checkpoint exposure, or representation behavior. A high score is not evidence of biological truth, universal transfer, or causal regulation. Check marker coherence, independent cohorts, appropriate baselines, and annotation quality under a prespecified validation plan.

These implementations perform exact neighbor search and are intended for modest teaching tables; they do not tune hyperparameters, calibrate probabilities, calculate integration metrics, test checkpoint pretraining exposure, or verify upstream model provenance. The R CSV outputs and Python JSON output are different report formats. Use only real vectors from a documented upstream model workflow for scientific analysis.

Official model sources

• scGPT official repository and its linked tutorials.
• Geneformer documentation and the official examples.