TL;DR
Are AI agents actually "reasoning" through biological data, or are they just recognizing the name of a drug and recalling its clinical trial outcome from memory? This paper introduces an "identity blinding" audit to show that many life-science benchmarks are inadvertently testing memorization rather than evidence-based reasoning, especially when the provided evidence is weak.
What Problem Does This Paper Address?
As AI agents are increasingly deployed for biological discovery, we need benchmarks to measure their ability to reason over complex evidence (like dossiers, sequences, or research papers). However, these benchmarks often include the names of real-world entities—genes, drugs, or diseases.
The authors identify a "Clever Hans" risk: an agent might achieve a high score not by reasoning over the provided evidence, but by recognizing the entity's name and retrieving the answer from its training data. This paper seeks to decouple identity (who the entity is) from evidence (the data provided about them) to see which one is actually driving the model's performance.
Methods
The researchers developed a protocol to audit benchmarks using several "contrast" conditions:
- BLIND: The entity names (e.g., "BRCA1") are replaced with opaque, "salted" tokens (e.g., "Gene_A"). This keeps the numeric and scientific evidence intact but hides the entity's identity.
- DELETE: The evidence block is removed entirely, leaving only the identity.
- PLACEBO: The evidence for one entity is replaced with evidence from a different, unrelated entity.
- RECALL: A direct probe where the model is asked for specific database records (like the number of clinical programs for a drug) without being given any evidence at all.
- Redaction Gradient: On the LitQA2 benchmark, the authors gradually redacted the supporting text (0%, 33%, 66%, 100%) while keeping entity names visible to see how the "blinding gap" grows as evidence disappears.
Datasets
The authors audited several prominent life-science benchmarks:
- PARROT (ours): A newly constructed clinical-outcome benchmark consisting of 286 target-indication instances, testing whether a drug program will advance past Phase II.
- LAB-Bench: Specifically the LitQA2 (literature snippets), SeqQA (biological sequences), and DbQA (database lookups) tasks.
- Biomni: Tasks involving gene screens and GWAS (Genome-Wide Association Study) candidate lists.
Key Insights and Ideas
- The "Blinding Cost" varies by task: If the evidence is a unique biological sequence, blinding the name doesn't matter because the sequence is the evidence. However, for tasks like GWAS, where the "evidence" is just a list of candidates, performance collapses when the name is hidden.
- Memorization is a major factor: In clinical benchmarks, the models could accurately "recall" the number of clinical programs for a specific drug/target pair without any evidence provided. This suggests the models are accessing a specific "snapshot" of a database they saw during training.
- Evidence deletion is a noisy signal: Simply deleting evidence (the "DELETE" condition) is a poor test because it changes the prompt's length, format, and semantic context all at once. Blinding identity is a much cleaner intervention for testing content use.
- Confidence is deceptive: Models were often just as confident when they were wrong (under the "false identity" condition) as when they were right, showing that "stated confidence" is a poor proxy for accuracy in these agents.
Key Results
- Clinical Benchmark: Accuracy dropped from 0.783 (named) to 0.531 (blinded), which is near the majority-class baseline of 0.521. This indicates the "dossier" provided was doing very little of the heavy lifting compared to the drug's name.
- LitQA2 Redaction: When 100% of the evidence passage was redacted, the gap between named and blinded performance reached +0.135, proving that the model relies on the name when the text is missing.
- GWAS Task: This showed the most extreme dependence, with a named accuracy of 0.742 plummeting to 0.082 when blinded (chance is 0.093).
- Recall Accuracy: The "no-evidence" recall probe achieved 0.818 accuracy, actually outperforming the model when it was given a full dossier.
Limitations and Future Work
The authors acknowledge that using opaque tokens (like "Gene_A") might inherently make reasoning harder for the model, meaning the "blinding cost" they measured might be an upper bound. Furthermore, the study was limited to two models from the same family (Claude).
The paper concludes that future life-science benchmarks must report identity-blinded results and retrieval telemetry to prove they are actually measuring reasoning. They suggest that the only way to truly test these agents is to use "temporally held-out" data—outcomes that occurred after the model's training cutoff.
Summary updated