When a language model writes a long, mechanistic scientific argument — say, a proposal for how to rescue a misfolded disease protein with a small molecule — it can fail in two very different ways. It can run out of things it actually knows and start speculating to fill the page, or it can make one wrong early claim and build everything downstream on that mistake. The first failure, "facts exhaustion," is what drives factual decay in everyday long-form generation: How Does Response Length Affect Long-Form Factuality showed that errors in long biographies do not snowball — precision simply slides as the model exhausts its verified knowledge (94.5% at 100 words down to 90.5% at 500). The second failure, error propagation, is what dominates multi-step reasoning: in multi-hop retrieval chains, a single failed intermediate step derails everything after it, and 63% of step-level failures in agent workflows trace back to upstream errors. Nobody knows which failure governs long-form scientific reasoning, because no existing evaluation can measure both at once.
We propose to settle this by building RescueBench: a benchmark of long-form protein-rescue hypotheses, evaluated by decomposing each essay into atomic claims (single, checkable propositions), arranging those claims on a directed acyclic graph — a diagram where each arrow means "this claim depends on that earlier claim" — mirroring the mechanistic chain from mutation, to destabilization, to degradation, to intervention. Protein rescue is the right domain for this question because the causal chains are genuinely multi-hop (a missense mutation raises folding free energy, which exposes an interface, which recruits the ubiquitin-proteasome system, which a corrector can counteract) and because ground truth lives in curated experimental numbers, not on the open web. Functional protein rescue is the application regime; long-form factuality supplies the atomic-claim machinery; the multi-hop gap defines the structural dependency we exploit; and LLM-as-a-judge is the instrument we must harden to make the measurement trustworthy.
Core Claim
The central scientific claim is that long-form multi-hop scientific reasoning is governed by error propagation, not facts exhaustion — the opposite of general-domain long-form generation — and that this difference is invisible to every existing evaluator: end-to-end LLM judges miss it because they score the whole essay, and current atomic-fact evaluators miss it because they treat claims as an unordered list. Detecting propagation requires both decomposition and dependency structure, and judging the claims requires grounding in typed physical evidence rather than web search.
Concretely, for each pair of claims where a child depends on a parent in the graph, we estimate
$$\Pi \;=\; \frac{P(\text{child unsupported} \mid \text{parent unsupported})}{P(\text{child unsupported} \mid \text{parent supported})},$$
where "unsupported" means the claim is contradicted by, or absent from, the curated evidence base. $\Pi \approx 1$ means errors are independent — the exhaustion regime BAFE found in biographies. $\Pi \gg 1$ means errors snowball through the dependency graph. The form is a simple conditional risk ratio; it is estimable per model and per essay length.
What We Build
The benchmark consists of three parts. First, a prompt set: roughly 300 open-ended prompts asking frontier and mid-size LLMs to write mechanistic rescue hypotheses (at controlled lengths of 200, 500, and 1000 words) for well-studied destabilized proteins — CFTR F508del, the FLCN variants behind Birt-Hogg-Dubé syndrome, lysosomal enzymes in Fabry, Pompe, and Gaucher disease, and NPC1 mutants. Second, a curated evidence base assembled from full texts and Supplemental material of the primary experimental literature, deliberately going beyond abstracts: Affinage showed that abstract-only pipelines achieve excellent faithfulness (97.95% supported claims, 0.029% contradictions) but structurally miss exact thermodynamic values, so we ingest full text for the biophysical anchors. The evidence base also ingests the small-scale direct measurements from VenusMutHub — 905 datasets across 527 proteins of measured $\Delta\Delta G$, $\Delta T_m$, $k_{cat}$, and binding affinities — precisely because that paper showed deep-mutational-scanning surrogates fail on direct data (the best stability predictor, MIF, reaches only 0.369 Spearman), making these numbers the scarce, high-value facts a model must either retrieve or hallucinate. Third, a gold-label set: every atomic claim in a 100-essay development subset is typed (thermodynamic, structural/interface, proteostasis-pathway, corrector-mechanism, empirical-magnitude) and labeled supported, contradicted, or out-of-evidence by two domain experts, with inter-annotator agreement reported.
Evaluation Protocol
Each generated essay is decomposed into atomic claims following the SAFE recipe from Long-form factuality in large language models, but with two departures. Verification runs against the curated evidence base instead of Google Search, because that paper's own limitation analysis concedes web-search verification breaks down in expert domains where ground truth sits in databases and unindexed primary literature. And the claims are then linked into the mechanistic DAG and judged step-by-step in the style of AgentEval, which found step-level DAG evaluation reaches 0.89 failure-detection recall versus 0.41 for end-to-end judging of the same traces, with strong human agreement (Cohen's $\kappa$ = 0.84). We extend AgentEval's step rubrics with claim-type-specific physical checks: a thermodynamic claim must match a measured $\Delta\Delta G$ within tolerance, an empirical-magnitude claim must match a reported rescue percentage, and so on.
Because the judge is an LLM, we audit it. Following the protocol of The Silent Judge, we inject provenance and recency cues ("reported by an EXPERT lab, 2025") into claims and measure the verdict shift rate; that paper found GPT-4o shifts +30% on recency cues with a 0% cue acknowledgment rate — the justifications never mention the cue that moved them. Our hypothesis is that evidence-anchored, typed verification is structurally resistant to these cues because the verdict is tied to a number in the evidence base, not to the prose around it; if the shift persists, that is itself a finding about judge hardening.
Research Program
The program has four experiments. The first and central one measures $\Pi$ across models and lengths on RescueBench, with a matched control on LongFact prompts from Long-form factuality in large language models to reproduce the BAFE no-propagation result under our own metric — this control rules out the objection that any propagation we find is an artifact of our measurement rather than the domain. The second experiment varies the generator: single-pass retrieval-augmented generation versus a CoRAG-style iterative chain from Chain-of-Retrieval Augmented Generation adapted to our evidence base, where intermediate sub-queries (e.g., "which interface does this mutation destabilize?") unlock later retrieval keys (matching correctors). CoRAG raised MuSiQue retrieval recall from 29.0% to 47.1% and beat baselines by over 10 exact-match points on general multi-hop QA; we test whether that gain transfers when the hops are biophysical. The third experiment is the judge bias audit described above. The fourth is a retrospective validity check with no lab work: we take historically measured rescue outcomes — c407 recovering approximately 54% of wild-type CFTR activity at 10 µM in HeLa cells, and the G1 + VX-809 combination restoring up to 45% versus roughly 8.5% for VX-809 alone in primary nasal cells, both from New insights into structure and function of bis-phosphinic acid derivatives and implications for CFTR modulation; wild-type FLCN half-life above 8 hours versus 3–4 hours for pathogenic variants, with destabilization of as little as 3 kcal/mol triggering aggressive degradation, from Folliculin variants linked to Birt-Hogg-Dubé syndrome are targeted for proteasomal degradation — and ask whether the judge's rescue-plausibility scores rank these interventions in the correct order. A judge that cannot reproduce the known ordering of measured rescue magnitudes cannot be trusted on novel hypotheses. For NPC1 proteostatic rescue we include the assay literature from Mutant induced neurons and humanized mice enable identification of Niemann-Pick type C1 as prompts, while noting that no reported rescue magnitude was found on Lacuna for it, so those essays are scored only on mechanistic claims, not empirical magnitude.
Baselines
The naive reference point is an unassisted end-to-end LLM judge: one strong model (e.g., GPT-4o) scores each whole essay against a rubric covering mechanistic soundness and factual accuracy, with no decomposition and no evidence base. This is the default practice and the floor to beat; AgentEval reports that this style of monolithic evaluation achieves failure-detection recall of only 0.41 on multi-step traces, and The Silent Judge shows it is systematically swayed by superficial cues it never discloses. We run it on identical essays with the same judge model and budget we use for our method, and we count ourselves as beating it only if our step-level protocol achieves materially higher claim-level failure-detection recall against the expert gold labels.
The strongest existing competitor is the SAFE/BAFE atomic-fact pipeline: decompose the essay into atomic claims, then verify each claim with Wikipedia lookup followed by web search, exactly as in Long-form factuality in large language models and How Does Response Length Affect Long-Form Factuality. BAFE reports 89.31% agreement with human annotators on general topics, so it is a genuinely strong method, not a strawman. We run it with the same claim decomposition module, the same judge model, and a matched per-essay verification budget (SAFE costs about $0.19 per response). The comparison is fair because the only difference is the evidence source and the dependency structure. We would count as beating it if our protocol achieves higher claim-verification accuracy against the expert gold labels, and we expect the margin to come precisely from its documented weakness: measured $\Delta\Delta G$ values, interface contacts, and rescue percentages from primary literature do not live on surface web pages, so web verification should default to "unverifiable" or, worse, to confident wrong verdicts. If web search turns out to recover these numbers well, that result would seriously undercut our motivation and we would report it as such.
Two further comparisons isolate the ingredients. To separate generation from evaluation, we run a CoRAG-style iterative retriever from Chain-of-Retrieval Augmented Generation — that paper reports 47.1% retrieval recall on MuSiQue versus 29.0% for single-step — as the generator under both our evaluator and the SAFE/BAFE evaluator. To isolate the DAG ingredient, we ablate our own judge down to a flat, unordered atomic-claim list scored with the same typed evidence (a BAFE-style list with physics grounding but no dependency edges): if $\Pi$ measured on the DAG predicts downstream error patterns and essay-level correctness better than flat-list precision does, the dependency structure is earning its place; if not, the simpler flat version wins and we should say so.
Why not just…?
Why not just fine-tune a specialized protein language model instead of building a retrieval-and-audit pipeline? Specialized models predict mutation effects, but they do not write or check long mechanistic arguments. ProtQueSt already compared the two approaches head-to-head for protein function annotation and found retrieval-augmented generation beat fine-tuned end-to-end models (BioT5+, Prot2Text-V2) by a +37% jump in Entity-BLEU (48.79), because protein function is query-dependent and static weights cannot flex with the query. Worse for the fine-tuning route, VenusMutHub shows the best specialized predictors reach only 0.369 Spearman on directly measured stability — too unreliable to serve as the ground truth our claims would be checked against. Fine-tuned models also give no auditable trace, which is the whole point here.
Why not just use single-pass retrieve-then-generate over the evidence base? Single-pass retrieval cannot know what to search for until intermediate deductions are made: you cannot retrieve the right corrector until you have established which interface the mutation destabilizes, and that fact is not in the query. Chain-of-Retrieval Augmented Generation quantifies this bottleneck — single-step retrieval reaches 29.0% recall on MuSiQue versus 47.1% for iterative chains. We include single-pass RAG as a generator condition rather than arguing around it, so the experiment itself shows whether the multi-hop structure of rescue reasoning (e.g., the dual-corrector synergy documented for CFTR in New insights into structure and function of bis-phosphinic acid derivatives) actually demands iterative retrieval in this domain.
Why not just prompt GPT-4o with a good rubric to score the whole essay? This is the naive baseline above, and the evidence against it is direct: end-to-end evaluation detects only 41% of step failures where step-level DAG evaluation detects 89% on the same traces (AgentEval), monolithic judges are misled by fluent prose hiding factual errors (Long-form factuality in large language models), and they silently shift verdicts by up to +30% on recency cues while acknowledging those cues 0% of the time (The Silent Judge). More fundamentally for this proposal, an end-to-end score cannot measure $\Pi$ at all — it has no notion of which claim depends on which — so it cannot answer the exhaustion-versus-propagation question even in principle.
Why not just ingest full-text papers and let the judge read everything? Affinage tested full-text ingestion directly: it increased extracted findings by only 16% at 15 times the cost, without improving summary quality, while its decoupled extract-then-synthesize design kept the contradiction rate at 0.029%. We adopt its decoupling — extraction separate from synthesis — but add what it lacks: full-text ingestion only for the narrow biophysical anchor values, plus typed verification and dependency structure, since Affinage is static, single-pass, and wild-type-only, with no notion of mutation destabilization or rescue.
Decisive Checks
Three results would make the case. First, the control experiment reproduces the BAFE no-propagation finding on LongFact under our metric ($\Pi \approx 1$), while RescueBench shows $\Pi$ substantially above 1 across models — establishing that the regime shift is real and domain-driven. Second, the evidence-grounded step-level judge beats both the end-to-end rubric judge and the web-search atomic pipeline on claim-level accuracy against expert gold labels, while showing near-zero verdict shift under injected provenance and recency cues. Third, the retrospective ranking check: judge scores order the historically measured interventions correctly (G1 + VX-809 above VX-809 alone, c407's ~54% recovery placed appropriately relative to combination therapy, FLCN variant half-lives ranked below wild type), with a Spearman correlation against measured rescue magnitudes reported openly. If the judge fails the ranking check, its scores should not be used to prioritize wet-lab work — and we would report that honestly, because a benchmark whose evaluator cannot reproduce known biology is a measurement device, not a discovery device. As a follow-on beyond this proposal, the highest- and lowest-scoring novel hypotheses from RescueBench would be handed to the programmable-stabilization toolkit of Programmable Protein Stabilization with Language Model-Derived Peptide Guides, which engineers deubiquitinase fusions to rescue undruggable targets but currently contains no text-based reasoning or automated evaluation — closing the loop between audited hypothesis generation and functional rescue at the bench.
References
- How Does Response Length Affect Long-Form Factuality
- Affinage: Genome-Scale Mechanistic Gene Annotation from the Published Literature
- VenusMutHub: A systematic evaluation of protein mutation effect predictors on small-scale experimental data
- Long-form factuality in large language models
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Chain-of-Retrieval Augmented Generation
- New insights into structure and function of bis-phosphinic acid derivatives and implications for CFTR modulation
- Folliculin variants linked to Birt-Hogg-Dubé syndrome are targeted for proteasomal degradation
- Mutant induced neurons and humanized mice enable identification of Niemann-Pick type C1 proteostatic therapies
- ProtQueSt: Query-Conditioned Retrieval-Augmented Generation for Protein Function Annotation
- Programmable Protein Stabilization with Language Model-Derived Peptide Guides