SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents

Brian T. Weston, Alondra Torres Contreras, Aldair E. GongoraarXiv 2026 · arXiv

TL;DR

SciResearch-Eval is a new benchmark designed to test "scientific judgment" in LLM agents—the ability to recognize when a simulator is flawed and adjust experimental strategy accordingly. While most current models are "confidently wrong" when faced with missing physics, the study reveals that top-tier models like GPT-6 Astra and Claude Fable 5.2 succeed by designing experiments that stay within the simulator's valid regime.

What Problem Does This Paper Address?

In the rush to build autonomous laboratory agents, most evaluations focus on outcomes: did the agent reproduce a number or find a law? However, real-world science often involves working with "leaky" abstractions—simulators that are under-resolved or physically incomplete.

The authors argue that a "good fit" to data is not evidence of a correct answer. An agent might fit a curve perfectly using a flawed model, resulting in physical parameters (like thermal conductivity) that are confidently incorrect. The core challenge addressed here is scientific judgment: the ability to detect systematic misfits (residuals that aren't just random noise), realize the tool is inadequate, and either widen uncertainty intervals or design "gentle" experiments where the flaws don't matter.

Methods

The researchers built an interactive environment centered on a classic lab problem: measuring a material's thermal properties.

  • The Setup: An agent is given a sample and a 1D heat-conduction simulator. It must design up to four experiments (choosing heater power, duration, and sensor placement) to estimate conductivity ($k$), heat capacity ($\rho c$), and heat loss ($h$).
  • The "Trap": Unknown to the agent, the "real" sample might have hidden complexities—like a glued joint, temperature-dependent properties, or side-wall heat loss—that the provided simulator cannot model.
  • The Clue: Agents can check for "patterned misfits." If the simulator is missing physics, the residuals between the model and the data will show a non-random structure.
Figure 1: The task, shown on a development task (no held-out answer is disclosed); red and blue mark the hot front and cooler rear, and magenta what is hidden from the agent. (a) A plate of unknown material is heated on its front face and loses heat to the air at its rear. The agent sees one experiment, may design four more (heater power and shape, duration, thermometer positions), and may run and fit a simulator of a one-piece plate at any resolution. It reports the conductivity k (how easily heat flows through the material), the heat capacity ρc (how much heat it stores) and the heat-loss coefficient h (how fast it loses heat to the air), each with a 95% range, scored against the exact truth. Hidden from the agent, this sample has a glued joint that the simulator lacks. (b) The simulator’s best fit looks near-perfect, yet its conductivity is 12% low; the misfit (bottom) is patterned, up to about 25 times the noise: the clue that physics is missing. (c) The evaluation pipeline. (d) The hidden flaw in each task family (described in Table 2); “basic version” in Figure 3 is missing physics with nothing else wrong.
  • Scoring: Instead of just point estimates, agents must report 95% confidence intervals. These are scored using a "proper interval score" that heavily penalizes being "confidently wrong" (narrow intervals that miss the truth).

Datasets

The paper introduces the SciResearch-Eval benchmark, consisting of:

  • 93 tasks in a held-out split, categorized into six "task families" (e.g., weak data, coarse solvers, missing physics).
  • 2,511 episodes generated by evaluating 9 different models (including GPT-6 Astra, the Claude 5.5 family, and open-weight models like Qwen and Gemma).
  • Exact ground truth: Because the "real" sample is itself a high-fidelity simulation, the true physical properties are known perfectly for scoring.

Key Insights and Ideas

  • A Good Fit is a Trap: Every model tested could produce "excellent-looking" fits. Without scientific judgment, they reported these fits as high-confidence truths, even when the underlying parameters were off by 10-20%.
  • Diagnosis $\neq$ Correction: Interestingly, many models (like Claude 5.5 Opus) could correctly name the missing physics (e.g., "there is likely a contact resistance") but still failed to provide accurate intervals. Diagnosis is a linguistic skill; correction is an experimental one.
  • The "Gentle Heating" Strategy: The most successful agents (Astra and Fable 5.2) didn't just complain about the simulator; they worked around it. By running low-power, short-duration experiments, they minimized the effects of missing physics (like temperature-dependent $k$), making the simple simulator "effectively" correct.
  • Reasoning Effort vs. Latent Skill: Claude 5.5 models failed at default settings but reached parity with Astra when forced to use "maximum reasoning" (writing 13–31x more tokens). This suggests the capability is latent but not prioritized at lower "effort" levels.

Key Results

  • Calibration Gap: At default settings, most models' 95% intervals contained the truth only 20%–62% of the time. In contrast, GPT-6 Astra (94%) and Claude Fable 5.2 (99%) remained highly calibrated.
  • The Cost of Thinking: Claude 5.5 Opus improved its interval score more than tenfold at maximum reasoning, but at a massive token cost.
  • Open-Weight Struggles: Models like Qwen3.8-27B and Gemma 4 31B showed almost no improvement with increased reasoning effort, remaining confidently wrong on physics-trap tasks.
Model coverage across different task families. Shaded areas represent tasks where the simulator is missing real physics.

Limitations and Future Work

  • Domain Specificity: The current benchmark is limited to thermal physics. The authors invite contributions to extend this to other domains like chemistry or fluid dynamics.
  • Human Baselines: The study lacks a direct comparison to human scientists, though a "play-it-yourself" interface has been released to collect this data.
  • Fixed Budgets: The agent is limited to four experiments; it's unclear if a larger budget would allow weaker models to "stumble" into the correct strategy.

The authors have released the environment, graders, and harness to encourage the development of agents that don't just fit data, but actually judge it.

Summary updated

Abstract

We introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval. To our knowledge it is the first benchmark to combine agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated intervals against exact ground truth. On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps. When the simulator is missing physics, most report wrong values while stating they are 95% sure: their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. At default effort the failing Claude 5.5 models rarely diagnose the missing physics (Opus in 35% of episodes, against 66% for Astra), and even when they do, their ranges often miss; Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause. Asking for wider intervals does not help; stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves. A pre-registered effort ablation shows that the correction is latent in the Claude 5.5 models: at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short. We release the environment, graders and harness.
BibTeX
@misc{weston2026sciresearch,
  author = {Brian T. Weston and Alondra Torres Contreras and Aldair E. Gongora},
  title = {{SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents}},
  year = {2026},
  eprint = {2611.21541},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21541},
}
Figure