When Should an AI Scientist Stop? Verifiable Experiment Steering and Refusal for Autonomous Discovery

N. G. Shah, Manglam Kartik

TL;DR

This paper introduces CARTOGRAPH, a novel "verification layer" for AI scientists that goes beyond simply proposing experiments. It enables AI systems to intelligently select the most informative experiments, resolve when a question is answered, and crucially, refuse to make claims when the underlying scientific model is inadequate. This framework significantly improves the reliability and trustworthiness of autonomous scientific discovery.

What Problem Does This Paper Address?

Current AI systems are increasingly capable of proposing and even executing scientific experiments. However, a critical gap exists: these systems often lack the ability to know when to stop, when they've found a reliable answer, or more importantly, when their underlying scientific model is fundamentally flawed and their claims are likely to be wrong. This is particularly problematic in high-stakes domains like drug discovery or materials science, where incorrect conclusions can have serious consequences. The paper addresses this by developing a principled way for AI scientists to verify their own findings and refuse to overclaim when evidence suggests structural inadequacy in their knowledge base.

Methods

The core of CARTOGRAPH is a three-part decision-making process:

  • Select: This component aims to choose the experiment that will most effectively reduce scientific ambiguity. It leverages concepts from Bayesian experimental design, specifically focusing on the "unresolved subspace" – the dimensions of the scientific question that current experiments haven't clarified. The paper proposes CARTOGRAPH-A, an A-optimal rule that is more precise than simpler projection-based methods.
  • Resolve: This is a straightforward "are we done?" signal. Ambiguity is considered resolved when the unresolved subspace shrinks to zero (in an exact setting) or when the accumulated information is sufficient to overcome a predefined threshold.
  • Refuse: This is the most novel contribution. Instead of just identifying a "best" model, CARTOGRAPH uses residual analysis to detect when the current library of scientific models is structurally inadequate. If the model's predictions don't align with observed data (i.e., residuals are too large), the system can refuse its tentative identification and flag potential library inadequacy. This is a "guardrail" against overconfidence.

The framework is built upon a "local linear-Gaussian bridge," meaning it assumes local linearity and Gaussian noise for tractability, but its theoretical underpinnings connect to classical methods like Fisher information and Bayesian experimental design.

Datasets

The paper evaluates CARTOGRAPH on several existing datasets and benchmarks:

  • Structured Nonlinear Cascade: A synthetic benchmark designed to test the "select" component across varying mechanism dimensions.
  • Pharmacokinetic (PK) Divergence Benchmark: A real-world dataset involving oral compartmental models to test the system's ability to handle low-dimensional problems.
  • EPA CvTdb Pharmacokinetic Time Series: Publicly available data used to assess the framework's feasibility in a real-world setting.
  • A-Lab Autonomous Materials System: A retrospective audit of published claims from a real autonomous materials discovery system.

Key Insights and Ideas

  • Refusal is as important as discovery: The most striking insight is the necessity of a "refuse" mechanism. Simply finding something isn't enough; knowing when not to claim something is crucial for scientific integrity.
  • Revocation is key to honesty: CARTOGRAPH can tentatively identify a model and then revoke that identification if subsequent data reveals structural misfit. This dynamic "honesty" is vital for avoiding false positives.
  • Unresolved subspace is the core: The concept of the "unresolved subspace" provides a unified way to think about selecting experiments and detecting when a question is answered.
  • Local approximations can be powerful: The paper demonstrates that even with local linear-Gaussian assumptions, the framework can achieve strong empirical results and connect to classical BOED theory.
  • Disagreement is a useful signal, but not enough: While disagreement between models is a useful indicator, CARTOGRAPH's A-optimal selection and residual-based refusal offer significant advantages.

Key Results

  • Stronger Selection: In a structured cascade benchmark, CARTOGRAPH-A significantly outperformed raw projection and disagreement-based methods in selecting the "hidden-best" experiment, achieving a 65% hit rate at d=8 compared to 2% for baselines (p < 10^-21).
  • Auditable Refusal: In a benchmark with out-of-library pharmacokinetic mechanisms, CARTOGRAPH successfully identified and then revoked tentative claims as residuals grew, demonstrating its refusal capability.
  • Real-world Audit Success: A retrospective audit of 40 claims from the A-Lab autonomous materials system flagged all 4 claims later deemed inconclusive by manual re-analysis, while correctly passing 32/36 confirmed claims.
  • Low-dimensional Near-Ties: In low-dimensional PK settings, CARTOGRAPH showed near-ties with disagreement, consistent with theoretical predictions, highlighting its ability to accurately predict performance limits.
  • Efficiency: The framework is computationally efficient, with per-step runtimes remaining in the millisecond scale.

Limitations and Future Work

  • Local Approximations: The theoretical guarantees rely on local linear-Gaussian assumptions. The authors acknowledge that performance may degrade in highly non-linear regimes or with heavy-tailed noise. Future work could explore more robust noise modeling and non-linear extensions.
  • Calibration: The performance of the "refuse" mechanism depends on the calibration of the threshold $\delta$. While the paper provides a method for calibration, it's sensitive to the size and representativeness of the calibration data.
  • Library Completeness: The "refuse" mechanism is designed to detect inadequacy in the current library. It doesn't inherently suggest what the correct model should be, only that the existing ones are insufficient.
  • LLM Integration: While the paper demonstrates an LLM-in-the-loop setup, the LLM's role is primarily candidate generation. Further research could explore deeper integration of LLMs in the decision-making process.

Summary updated

Abstract

We present CARTOGRAPH, a verification layer for AI scientists that couples unresolved-subspace experiment steering (select), explicit ambiguity closure (resolve), and residual-based library inadequacy detection (refuse). Under a local linear-Gaussian bridge, raw unresolved projection is the isotropic unresolved Fisher-information trace, while CARTOGRAPH-A is the exact unresolved A-optimal rule; closed-form EIG and Box-Hill arise as local comparators rather than global equivalents. Across five testbeds, CARTOGRAPH-A beats raw projection 129W/0T/15L at d = 8 (p &lt; 10^-21) in a replicated structured cascade. More distinctively, the framework tentatively identifies three out-of-library pharmacokinetic mechanisms and then revokes those identifications as residuals expose structural misfit, while one perturbed in-library control stays identified throughout. In low-dimensional pharmacokinetic and filtered EPA settings, near-ties against disagreement are predicted by theory and observed. Finally, in a retrospective audit of 40 positive claims from the published A-Lab autonomous materials system, the refuse guard flags all 4 claims later marked inconclusive under manual reanalysis while passing 32/36 confirmed claims. Code is available at https://github.com/ai4science-boed/cartograph.git
BibTeX
@inproceedings{shah2026when,
  author = {N. G. Shah and Manglam Kartik},
  title = {{When Should an AI Scientist Stop? Verifiable Experiment Steering and Refusal for Autonomous Discovery}},
  booktitle = {AI4Science @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=DdDTacsJnr},
}

Loading related content...

Figure