Language-model agents now promise to let a biologist type "find the differentiation lineages in this dataset" and get a finished single-cell analysis back. The evidence that they work comes from public datasets whose answers are written down all over the literature these models were trained on. A high score can mean the agent analyzed the expression matrix. It can also mean the agent recognized familiar gene names and recalled what papers say about them.
We propose a benchmark generator that separates these two explanations. It synthesizes single-cell datasets around a hidden developmental "skeleton" with exactly known shape: a set of branch points, dead-end terminal states, cycles and disconnected populations. It gives the genes names the model has never seen, and it can add controlled distortions that make a naive plot show a branch that does not exist. Because the answer is a small set of exact topological counts fixed by construction, scoring is deterministic and needs neither expert annotation nor a language model as judge. Because every instance is freshly generated, the test set cannot leak into pretraining.
Core Claim
Single-cell structure-discovery tasks can be made contamination-proof and exactly gradable at the same time. The trick is to generate data from a latent trajectory graph whose singular points, meaning branch hubs, terminal dead ends and loops, are the ground truth. The claim that follows is empirical and falsifiable. Current agents such as CellAgent and zero-shot LLM annotators of the kind studied in SOAR will show two things. First, they will lose accuracy when gene names are remapped. Second, they will lose much more accuracy when the dataset contains projection artifacts that create false branches. Classical algorithmic pipelines will degrade gracefully under both conditions.
If the remapping gap is near zero, the tension the literature leaves open is resolved in SOAR's favor. That tension is between SOAR's "latent biological intuition" and the memorization evidence from A²RBench and DyVal. If the gap is large, deployed agents are being credited for recall.
How the Four Directions Fit
The four directions do not carry equal weight here.
Contamination is the problem. Two papers supply the method: DyVal, which generates reasoning problems from random directed acyclic graphs with tunable depth and width, and A²RBench, which accepts generated tasks only if they pass programmatic checks and which showed a 17.7% drop when familiar symbols are remapped. Both work only on abstract puzzles. We carry their generate-and-verify approach into a scientific workflow, which the file identifies as their main open limitation.
Bioinformatics accessibility is the application. CellAgent reports over 96% execution success and an annotation score of 0.85 against scGPT's 0.77 on 64 fixed public datasets. It validates its outputs with an LLM evaluator that inspects plots. SOAR serializes marker genes into prompts and reports a GPT-4o BLEU-1 of 62.85. Neither controls for whether canonical marker-to-cell-type associations were simply recalled.
Singularity theory supplies the ground-truth vocabulary and the difficulty knob. Global Topology of 3D Symmetric Tensor Fields summarizes a continuous field by its degeneracies, classifying them as "wedges" where structure terminates and "trisectors" that split three ways, and characterizing regions with Betti numbers. Betti numbers are counts of connected pieces ($\beta_0$) and independent loops ($\beta_1$). We use the same idea for trajectories: a lineage tree's branch hubs play the role of trisectors, and its terminal fates play the role of wedges.
Geometry of Developable Surfaces of Frenet Type Framed Base Curves from the Singularity Theory Viewpoint classifies the generic singularities of a smooth map into canonical forms such as cuspidal edges and swallowtails. We borrow the related fact that when a smooth trajectory is squeezed into a low-dimensional plot, folds and cusps appear generically. These projection singularities can make a single lineage look like it branches or crosses itself. Planting them at a controlled rate gives a principled hardness axis. It separates an agent that tests structure in the full expression space from one that reads the two-dimensional plot. We are not advancing singularity theory. We use its classification to decide which distortions are generic and therefore realistic.
Static analysis is a secondary diagnostic, not a scalability contribution. Lightweight data-flow tracking over agent-generated scripts, in the staged cheap-filter-first spirit of cascade, asks one question: does the output label depend on the input matrix, or on constants such as a hard-coded marker list? As discussed below, re-execution on remapped data may answer the same question more cheaply. We treat the two as competitors to be compared rather than assuming static analysis is needed.
Research Program
The generator samples a latent trajectory graph $G$ with a chosen number of components, branch hubs, terminal states and cycles. It embeds $G$ as smooth curves in a latent space and maps latent position to gene expression through random smooth response functions. It then adds count noise and dropout at levels matched to real 10X Chromium data statistics, drawn from a reference atlas.
A fold/cusp module deliberately chooses response functions whose composition with standard dimensionality reduction produces a specified number of generic projection singularities. The generator verifies each planted singularity by locating where the Jacobian of the latent-to-plot map loses rank, and it verifies that the true $\beta_0$ and $\beta_1$ of $G$ are unchanged. Following A²RBench's acceptance rule, an instance is admitted only if an oracle given the latent coordinates recovers $G$ exactly.
A synthetic "reference card" assigns fictional marker genes to fictional cell types. Annotation tasks then test whether an agent uses supplied knowledge rather than recalled knowledge. Each instance is emitted in two versions:
- Canonical: real human gene symbols mapped onto the synthetic programs.
- Remapped: novel symbols.
Tasks include counting lineages and terminal fates, recovering the lineage graph, locating branch hubs, detecting cycles, naming the genes that drive a branch, and annotating with the reference card. Difficulty is swept along five axes, following DyVal's tiered design:
- graph depth
- branch count
- noise
- cusp count
- remapping
Baselines
Naive reference. A fixed default analysis script: standard normalization, clustering and graph abstraction with default parameters, run with no adaptation. This is what a novice would get by copying a tutorial. We also report a majority-count guess, the most frequent true branch count in the generator's distribution. Any agent must beat both to claim it adds analysis rather than boilerplate.
Strongest algorithmic reference. The best clustering and trajectory methods available through DANCE. DANCE standardizes 32 methods on 21 datasets and reports, for example, graph-sc improving ARI from 0.78 to 0.82 on mouse embryonic stem cells. These methods are gene-name-blind by construction, so they set the ceiling a data-reading agent should approach under remapping. Each DANCE method gets the same tuning budget as the agents: a fixed number of configuration trials, selected on a held-out generator split, never on test instances.
Strongest agent. CellAgent with its Planner, Executor and Evaluator. Its reported figures on its own static suite are 96% execution success and a 0.85 annotation score. We also test a SOAR-style prompted annotator; SOAR reports GPT-4o BLEU-1 62.85 in zero-shot RNA annotation. Both run on identical instances with an identical token and wall-clock budget. They run both with the proprietary backbone the papers used and with open-weight backbones, which addresses the proprietary-dependence limitation the file notes.
Beating these baselines is not the goal, since this is a measurement proposal. The decisive quantity for an agent is its remapping gap and its cusp slope. Specifically, the proposal succeeds if it produces a statistically reliable, reproducible difference between canonical and remapped accuracy for LLM agents that is absent for DANCE methods. Across generator seeds, the confidence interval on that difference must exclude zero.
Key-ingredient ablations. The generator is run with the cusp module off, so difficulty comes only from noise and depth. It is also run with remapping off. These show whether planted projection singularities and name remapping each carry diagnostic signal the other lacks.
Why not just…?
Why not just rename genes in existing real datasets? This is the most serious alternative. For the narrow annotation question it may be enough, and we will run it: PBMC and the other public CellAgent datasets with remapped symbols. If the remapping gap on real data matches the gap on synthetic data, the synthetic generator is unnecessary for annotation, and we will say so. Renaming cannot, however, supply ground truth for structure discovery. The true number of lineages or cycles in a real tissue is itself an expert judgment, and expert judgment is circular when the expert literature is what the model memorized. Planted topology is what makes lineage-count and branch-hub tasks exactly gradable.
Why not just use an LLM judge or plot inspection, as CellAgent's Evaluator does? Two findings in the file argue against relying on it. BioSkillSafety shows a strong frontier model blocking only 20.5% of domain-specific attacks, and DyVal finds 33.8% of GPT-4 failures are unsubstantiated responses. More directly, the cusp module is designed to fool plot reading. We will score the Evaluator's verdicts against exact invariants. If its agreement stays high as the cusp count rises, the alternative holds and exact scoring adds less than we claim.
Why not just use an existing single-cell simulator? We likely should use one as the expression backbone. No reported result was found on Lacuna for single-cell simulators, and none is cited in the file. The contribution here is the layer above the simulator: invariant-labeled task generation, generic-singularity hardness control, oracle-verified acceptance, paired remapping, and a fictional reference card.
Why not just re-run the agent's code on remapped data instead of statically analyzing it? This may work as well as static analysis. Re-execution directly shows whether the outputs depend on names. We run both methods, measure how often they agree when flagging hard-coded knowledge, and keep static analysis only if it catches cases that re-execution misses or costs meaningfully less.
Proposed Experiments
Validity (run first). Two checks establish that the generator itself is sound. First, at low noise the latent-space oracle and at least one DANCE trajectory method should recover $G$ nearly perfectly. Second, a small panel of human analysts, sized like CellAgent's 20-researcher user study, should find remapped synthetic instances indistinguishable in difficulty from canonical ones. If either check fails, the generator, not the agents, is being measured.
Main study. We sweep the five difficulty axes over roughly 500 instances per cell, following DyVal's per-dataset size. We report five metrics:
- exact-match accuracy on $\beta_0$, $\beta_1$, branch-hub count and terminal count
- graph edit distance between recovered and true lineage graphs
- annotation accuracy against the fictional reference card
- the remapping gap
- the slope of accuracy versus planted cusp count
Decisive checks. Three outcomes would be decisive:
- Recall-driven agents. If LLM agents show a large remapping gap while DANCE methods show none, their real-data scores overstate their data-reading ability.
- Plot-reading agents. If agent accuracy falls with cusp count faster than algorithmic accuracy, the agents are reading two-dimensional plots rather than testing structure in the full expression space.
- Synthetic-only artifact. If the remapping gap on renamed real data differs sharply from the synthetic gap, synthetic realism is the problem, and the generator's expression model must be revised before any conclusion is drawn.
References
- CellAgent: LLM-Driven Multi-Agent Framework for Natural Language-Based Single-Cell Analysis
- Single-Cell Omics Arena: A Benchmark Study for Large Language Models on Cell Type Annotation Using Single-Cell Data
- A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Global Topology of 3D Symmetric Tensor Fields
- Geometry of Developable Surfaces of Frenet Type Framed Base Curves from the Singularity Theory Viewpoint
- cascade: C Assertion Checker and Deductive Engine
- DANCE: a deep learning library and benchmark platform for single-cell analysis
- BioSkillSafety: A Systematic Benchmark for Evaluating Agent Skill Safety in Bioinformatics