Counterfactual Replay for Auditing Whether Variant-Interpreting Agents Read the Evidence

When a language-model agent reads a patient's exome data, queries ClinVar and the literature, and concludes that a variant is pathogenic, a correct answer does not show that the agent used what it retrieved. The model may simply be recalling a classification it saw during pretraining. This matters because public variant databases leak into training corpora, and the clinically important cases are the novel and reclassified variants where memory is wrong or missing. We propose to measure this directly. The agent runs inside a fully deterministic container. Each run is replayed with exactly one tool response changed so that the evidence points the other way, for example a functional-assay abstract whose result is flipped. Whether the decision flips with the evidence gives a clean, per-decision label: evidence-following or memory-following. We then train a cheap probe on the model's internal activations to predict that label from a single, unperturbed run, so the audit can be applied at deployment without paying for replays.

The pieces have distinct roles. Reproducible execution is the measuring instrument, because only a hermetic, version-pinned sandbox makes the replay identical up to the intervention. Whole-exome variant interpretation is the application, chosen because label leakage there is pervasive and costly. Interpretability supplies the amortized monitor. Dataset transparency supplies a leakage manifest that records, for every benchmark item, whether its answer was public before the model's training cutoff.

Core Claim

Our claim has two parts. First, counterfactual replay in a deterministic sandbox yields reliable per-decision labels of evidence reliance, which existing log-based audits cannot produce. Second, a linear probe on the agent's internal states, read at the step where the decision is made, predicts those labels on held-out genes well enough to replace most replays. If the second part fails, the first still stands on its own: a replay audit protocol, together with a leakage-stratified benchmark, would show how much of current agents' reported accuracy on variant interpretation comes from memory.

Relation to Prior Work

The closest system is SciAgent, from Audit-Grade Harness for Agent-driven Scientific Computation Workflows. It pairs pinned containers with append-only provenance logs and a "fresh-context" LLM verifier, and it caught agents fabricating data after tool failures. Its verifier reads only logs and artifacts, its confidence scores are uncalibrated, and it runs single recordings. Our proposal keeps its container discipline but uses the container differently: as an intervention device rather than as a record. Two variant-interpretation agents motivate the question without answering it. AIVARI Agent retrieves from OMIM and ClinVar and reports group sensitivity 0.956 on 300 private clinical cases. VarLitBench and VarLitAgent benchmarks the extraction of functional-assay evidence. Neither system asks whether its tools actually changed its answers.

The method comes from Unveiling Knowledge Utilization Mechanisms in LLM-based Retrieval-Augmented Generation. That work found a layer-wise "contestation" between memory and retrieved context in LLaMA-3 and Qwen-2.5. It also isolated "external-knowledge" neurons by their activation entropy: deactivating them with a gold passage cut F1 by about 11.36 points. Its evidence comes from short-form general question answering. We test whether such signals survive multi-step tool use in a specialist domain, and we supervise them with counterfactual labels instead of trusting them by construction. Investigating Layer Importance in Large Language Models constrains where we look. It found importance concentrated in a few early "cornerstone" layers on general benchmarks, which warns that probe signal may be layer-specific, so we sweep all layers rather than assuming a location.

The transparency motivation comes from three sources. Fell et al. showed that undocumented preprocessing choices collapsed reproduced pathology models, for example slide accuracy falling from 92% to 54.7%. MVP documents circularity between training and test sets in public variant databases. Zhu et al. found that 70% of audited agent benchmarks violated both task and outcome validity, and that a do-nothing agent scored 38% on τ-bench. We take from these the lesson that the benchmark itself needs a provenance record and a trivial-agent check.

Research Program

We first build a hermetic harness for an open-weight agent, with Llama-3 and Qwen-2.5 as the primary models since prior probing work used them. The harness uses greedy decoding, a pinned container image, and a tool layer that serves cached, versioned snapshots of ClinVar records, literature abstracts, and population-frequency lookups instead of live APIs. We make no claim of determinism without checking it. Before any experiment, we replay every trajectory unmodified and report the fraction that reproduce token-for-token. Any trajectory that does not reproduce is excluded and counted.

Next we define the intervention. For a decision $d$ made after tool output $e$, we construct a counterfactual $e'$ that, under the ACMG-style rule the task uses, should reverse $d$. In VarLitBench this means a PS3 abstract becomes BS3, meaning an assay showing damage is rewritten to show normal function. In ClinVar-backed tasks it means swapping a record's assertion and its supporting text. The replay is identical up to the step that returns $e'$. The label is 1 if the decision reverses and 0 if it holds. A second, closed-book run with tools disabled records what the model believes from memory. Together these sort every decision into one of four groups, according to whether evidence and memory agree and which one the agent followed. The informative group is the one where they disagree. We also run sham interventions that paraphrase $e$ without changing its meaning, to estimate how often decisions flip for no reason.

The leakage manifest attaches to each benchmark item the earliest public date of its classification or abstract, together with whether that date precedes the model's training cutoff and whether the variant was later reclassified. As a planned domain-shift slice, we add near-intronic variants at the +5/−5 positions that Zhang et al. identified as damaging splice hotspots in off-target exome capture. We hypothesize, without assuming, that these are less memorized than coding missense variants.

Finally we train probes. At the step where the agent commits to a decision, we read residual-stream activations at every layer, plus the activation entropy of the external-knowledge neurons as defined by Wang et al. We then train linear probes to predict the replay label. Splits are by gene, so the probe cannot learn gene identity, and we report results separately for each leakage stratum.

Baselines

The naive reference point predicts the base rate, labeling every decision evidence-following. We report it alongside a do-nothing agent that outputs the majority class without calling tools, following Zhu et al., to show how much benchmark accuracy is available without any evidence at all.

The strongest existing audit is SciAgent's fresh-context verifier. We reimplement it with the same open-weight model family and give it the full append-only log and artifacts for each trajectory. We ask it to judge whether the decision was grounded in the retrieved evidence and score its confidence by AUROC against the replay labels. Its prompts get the same tuning budget, on the same training genes, as the probe's hyperparameters. No reported result found on Lacuna for any verifier's accuracy at detecting parametric reliance, so this comparison is new measurement rather than a lookup.

For behavioral context, the published reference on VarLitBench is o4-mini's PS3-versus-BS3 direction classification at 0.979 F1 (VarLitBench). Our question is whether accuracy that high survives counterfactual replay. AIVARI's 0.956 sensitivity is on private data and cannot be rerun, so we cite it but do not compare against it.

Three ablations isolate the key ingredients. The first uses the closed-book proxy, labeling a decision memory-driven whenever the closed-book answer matches. The second uses the model's own token probability on the decision. The third uses only the entropy of the external-knowledge neurons, untrained, as Wang et al. defined it. A fourth ablation trains the same probe on labels from a non-hermetic harness, with live APIs and sampled decoding, to test whether determinism matters. The probe beats the baselines if, on held-out genes in the stratum where memory and evidence disagree, it exceeds the verifier's AUROC by a margin larger than the bootstrap 95% interval and stays calibrated, with expected calibration error under 0.1.

Why not just…?

Why not just run the counterfactual replay on every case and skip the probe? This is a real threat. If replays are cheap, the probe is a convenience rather than a necessity. Two things limit replays. A rigorous audit needs several interventions per decision, one per piece of evidence, plus shams, which multiplies cost. VarLitBench's agentic mode already takes over five minutes per variant and was limited to 20 variants for cost reasons (VarLitBench). We will report the probe's accuracy as a function of the number of replays it replaces. If it does no better than one replay, we will recommend replay alone and say so.

Why not just test on variants classified after the training cutoff? Temporal holdout measures memorization across a population, but it says nothing about which individual decisions on old variants were grounded. It also throws away most of the benchmark. We include it as one stratum of the leakage manifest, so it serves as a check on the probe rather than a substitute for it.

Why not just ask the model closed-book and treat a correct memory answer as leakage? When memory and evidence agree, a correct closed-book answer does not show that the tool-using run ignored the tools. This proxy is one of our ablations. Comparing it against replay labels shows directly how often it confuses "the model knew" with "the model didn't read."

Why not just use a stronger LLM verifier? SciAgent's authors concede that the verifier's scores are uncalibrated and that it can hallucinate an audit (Badhwar et al.). More fundamentally, logs look the same whether a model copied a ClinVar assertion or recalled it. The verifier comparison in the Baselines section tests this. If a strong verifier matches the probe, that is a useful negative result for interpretability.

Proposed Experiments and Decisive Checks

The first check is a precondition: the token-for-token reproduction rate of unmodified replays should be near-perfect. If not, labels are noisy and every later result is compromised, which is itself a finding about how containerized agent evaluation handles GPU nondeterminism.

The second experiment applies the replay audit to VarLitBench and to a ClinVar-derived classification task built from the cached snapshot. It reports the evidence-following rate for each leakage stratum and for the splice-hotspot slice. The finding that would change practice is a large gap between accuracy and evidence-following on pre-cutoff variants.

The third experiment is the probe against the baselines above, evaluated on held-out genes and on the out-of-domain splice slice.

The fourth is a causal check. Deactivating the external-knowledge neurons should lower the evidence-following rate more on decisions the probe scored as evidence-driven than on those it scored as memory-driven. If both groups drop equally, the probe is reading a correlate rather than the mechanism.

We will release the container image, the cached tool snapshots, the replay logs, and the leakage manifest, so that every number can be regenerated.