Catching Silent Failures in Multi-Slice Spatial Omics Agents

Language-model agents can now write and debug bioinformatics code until it runs. Running is not the same as being right. In spatial transcriptomics, which measures gene activity at known positions across a series of thin tissue sections, the worst failures make no noise. A registration step mirrors one section relative to its neighbors. A batch-correction step blends away a real boundary between cortical layers. A clustering step smooths fine layers into one blob. The code exits cleanly, the summary numbers look fine, and the result is wrong. Spotting these errors means comparing several sections against each other. That is exactly the multi-image reasoning that vision-language models (models that read images and text together) are measurably bad at.

This note proposes a critic agent for code-writing analysis agents. After each run it renders diagnostic images of adjacent tissue sections, judges cross-section consistency one pair at a time rather than all at once, and sends targeted repair requests back to the code-writing agent. The central question is not whether such a loop can be built. It is whether looking at images catches failures that cheap, label-free numerical checks miss. The note is built around a controlled failure-injection benchmark that can answer that question either way.

Core Claim

Existing analysis agents stop one step short. AutoBA plans multi-omic pipelines, generates scripts, and repairs them from error tracebacks. It reports 87.5% end-to-end execution success over 40 cases, but it never looks at an output. CellAgent adds an evaluator that inspects single diagnostic plots, such as one UMAP (a 2-D projection of cells) or one section's spatial map, with GPT-4o. It reports over 96% execution success and a 0.85 annotation score. It never compares one section against another. CP-Agent reasons over cell images under drug perturbations, but through a fixed pipeline that writes no code.

The new element here is narrower than "add vision." It is a cross-section critic whose judgments are broken into adjacent-pair comparisons, plus a test of whether those judgments carry information beyond label-free metrics.

The claim has two parts. First, there is a family of silent failures in multi-slice pipelines where label-free metrics either stay flat or improve while true layer recovery drops. The clearest case is over-correction: batch-mixing scores rise as real niche boundaries are erased, a tension the spatial integration literature names as an open problem. Second, a pairwise visual critic detects these failures reliably. A critic shown the whole stack in one call does not.

The second part follows from evidence about why vision models fail on many images. More Images, More Problems? shows that degradation tracks the total length of visual tokens rather than image count. Restricting attention to within-image tokens in deeper layers raised a 7B LLaVA-OV model on MuirBench from 41.7% to 51.3%. Cutting a 12-section comparison into overlapping pairs is the agent-level version of that remedy.

Research Program

The system has three roles. Separating the critic from the code writer is motivated by findings that a model debugging its own code reinforces its own mistakes (Srivastava et al.), and that a separate tester role helps (Self-collaboration Code Generation).

A planner turns the analysis request into steps: preprocessing, joint embedding, cross-section alignment, and domain clustering. It may wrap existing methods such as JADE or stMSA rather than invent new ones. An executor writes and repairs code from tracebacks, as AutoBA does. It also keeps a rollback buffer of the last pipeline state that passed review, borrowing the regression guard of Fang et al..

The critic receives a fixed rendering protocol. For each adjacent pair of sections it gets three separate images, in the same pixel frame after alignment:

  • the domain labels;
  • one reference marker gene;
  • an overlay of matched spots.

The images are provided as separate files, interleaved with short text describing each one. This follows ReMI, which found that models do worse on stitched collages and better when images are interleaved with text.

For each pair, the critic answers a short structured checklist in free text. Is the orientation consistent? Do layer boundaries continue across the pair? Are any domains present in one section and absent in its neighbor without explanation? Has any domain spread across the whole section?

An aggregator turns the pairwise verdicts into a localized repair request, for example "section 3 appears mirrored relative to 2 and 4." The executor then edits the code. The critic is also allowed to answer "cannot tell." MuirBench reports drops often exceeding 20% on unanswerable variants, so abstention is a known weakness and is scored separately.

The critic model is varied as a scientific factor, not fixed. Runs use a frontier proprietary model and an open 7B LLaVA-OV model, with and without the within-image attention masking from More Images, More Problems?. This shows whether the masking remedy carries over from natural images to tissue sections. That transfer is untested, since the multi-image benchmarks deliberately exclude biomedical serial images.

Failure-Injection Benchmark

Most of the evidence comes from a benchmark built on the human dorsolateral prefrontal cortex (DLPFC) Visium data: 12 sections with expert layer annotations. We take strong reference pipelines and inject known silent failures at controlled severities:

  • mirroring or rotating one section after alignment;
  • shifting one section's registration by a fraction of its width;
  • over-smoothing the clustering by inflating neighborhood size;
  • over-correcting batches with an aggressive mixing step;
  • swapping domain labels between two sections;
  • omitting spatial-graph construction so that clusters fragment.

Each corrupted run gets an exact ground-truth label: which failure, which section, and what severity. This makes detection measurable without hiring annotators. Every corruption is checked to exit with code zero, so traceback-based repair cannot see it by construction.

For each run we record the true layer ARI against the hidden annotations. ARI (adjusted Rand index) scores agreement with the expert layers, where 1 is perfect and 0 is chance. We also record every label-free metric an agent could compute without annotations:

  • iLISI, a batch-mixing score whose maximum for two batches is 2.0;
  • silhouette;
  • spatial autocorrelation of domains;
  • optimal-transport alignment cost;
  • per-section domain proportions.

The first analysis asks how often label-free metrics move in the wrong direction when ARI drops. If they rarely do, the visual critic has no reason to exist, and the project reports that.

For generalization, the same failures are injected into the mouse embryo MOSTA Stereo-seq series, a different platform and resolution. No reported quantitative result was found on Lacuna for multi-slice alignment on MOSTA, so results there are reported only relative to our own baselines.

Baselines

The do-nothing reference is the reference method run once with default settings and no agent. JADE reports a median DLPFC ARI of about 0.61–0.65 and iLISI of 1.98. stMSA reports an average ARI of 0.544. We rerun both on the same sections and preprocessing, so the comparison uses our reproduced numbers rather than the published ones. The published values serve only as sanity checks. SAINT reports an ARI of 0.75, but that is for single-section clustering with DNA sequence features, a different task. It is cited, not treated as a comparable target.

The traceback-only agent reproduces AutoBA's repair behavior inside our executor. It is the natural "agents already do this" competitor. On the injected benchmark it should detect nothing, and it serves as a floor for loop overhead.

The single-image critic reproduces CellAgent's evaluator: it inspects one rendered section at a time and may also see quantitative scores. It is the closest existing system. Beating it isolates the value of comparing across sections rather than the value of vision at all.

The strongest competitor, and the one the proposal must beat, is the metric-only critic. This is the same critic model in text-only form, given every label-free metric listed above, per section and per pair, and allowed the same repair budget.

There are two ablations. The full-stack critic receives all 12 sections in a single call, which isolates pairwise decomposition. The no-rollback variant isolates the regression guard.

All agentic variants share the same executor, the same critic model where relevant, the same budget of repair iterations and tokens, and the same prompt-tuning budget. Prompts are tuned on one DLPFC donor and evaluated on the other two.

Beating the metric-only critic is defined in advance. The pairwise visual critic must achieve higher detection AUROC (the probability it ranks a corrupted run above a clean one) on injected failures, and higher final ARI after repair, on held-out donors. Both margins must be larger than the spread across three repeat runs. It must also keep false alarms on clean runs comparable, so the loop does not "repair" good pipelines into worse ones.

Why not just…?

Why not just feed label-free metrics to the critic instead of images? This is the real threat, and the proposal is designed around it. Orientation errors, for instance, may be caught perfectly by optimal-transport cost, and if so, images add nothing for that failure. The case for images rests on over-correction and over-smoothing, where batch-mixing scores can rise while layers disappear. The literature names this tension but quantifies it nowhere in the file. The metric-only baseline and the per-failure breakdown settle it. Wherever metrics match vision, we recommend metrics.

Why not just tune JADE or stMSA directly and skip the agent? A hyperparameter search needs an objective. Without annotations, the only available objectives are the same label-free metrics, which inherits their blind spots. JADE also reports accuracy problems across 300µm section gaps for earlier aligners, and stMSA needs landmark-guided registration to handle flipped sections, so there is real configuration work an agent could absorb. Still, the comparison here is to search over the reference method's settings, scored by the best label-free metric, under the same compute budget. If that search matches the agent's final ARI, the agent's contribution reduces to detection, and we report it that way.

Why not just give one strong multimodal model every section at once, or tile them into a single image? ReMI reports that collages hurt performance. More Images, More Problems? shows that accuracy degrades with total visual token length, which tiling high-resolution tissue maps makes worse. But GPT-4o reaches 68.0% on MuirBench, so a frontier model might handle 12 low-resolution panels well enough. The full-stack ablation tests this directly. If it matches pairwise decomposition, the decomposition claim fails.

Decisive Checks

Three results decide the project.

The first is the per-failure detection table on held-out DLPFC donors, comparing the pairwise visual critic against the metric-only critic. If vision wins on no failure type, the visual loop is abandoned. What remains is the benchmark and a measured account of where label-free metrics mislead.

The second is final hidden-label ARI after closed-loop repair, compared with the reproduced JADE and stMSA defaults and with the metric-guided settings search. This shows whether better detection becomes better analyses.

The third is the pairwise-versus-full-stack gap, measured with and without within-image attention masking in the open 7B critic. This checks whether the dilution account from natural-image benchmarks holds on serial tissue. If it does, an open model can stand in for the proprietary dependency that CellAgent leaves open.

Repeating all three on MOSTA shows whether the findings survive a change of platform.