TL;DR
SciResearch-Eval is a new benchmark designed to test "scientific judgment" in LLM agents—the ability to recognize when a simulator is flawed and adjust experimental strategy accordingly. While most current models are "confidently wrong" when faced with missing physics, the study reveals that top-tier models like GPT-6 Astra and Claude Fable 5.2 succeed by designing experiments that stay within the simulator's valid regime.
What Problem Does This Paper Address?
In the rush to build autonomous laboratory agents, most evaluations focus on outcomes: did the agent reproduce a number or find a law? However, real-world science often involves working with "leaky" abstractions—simulators that are under-resolved or physically incomplete.
The authors argue that a "good fit" to data is not evidence of a correct answer. An agent might fit a curve perfectly using a flawed model, resulting in physical parameters (like thermal conductivity) that are confidently incorrect. The core challenge addressed here is scientific judgment: the ability to detect systematic misfits (residuals that aren't just random noise), realize the tool is inadequate, and either widen uncertainty intervals or design "gentle" experiments where the flaws don't matter.
Methods
The researchers built an interactive environment centered on a classic lab problem: measuring a material's thermal properties.
- The Setup: An agent is given a sample and a 1D heat-conduction simulator. It must design up to four experiments (choosing heater power, duration, and sensor placement) to estimate conductivity ($k$), heat capacity ($\rho c$), and heat loss ($h$).
- The "Trap": Unknown to the agent, the "real" sample might have hidden complexities—like a glued joint, temperature-dependent properties, or side-wall heat loss—that the provided simulator cannot model.
- The Clue: Agents can check for "patterned misfits." If the simulator is missing physics, the residuals between the model and the data will show a non-random structure.
- Scoring: Instead of just point estimates, agents must report 95% confidence intervals. These are scored using a "proper interval score" that heavily penalizes being "confidently wrong" (narrow intervals that miss the truth).
Datasets
The paper introduces the SciResearch-Eval benchmark, consisting of:
- 93 tasks in a held-out split, categorized into six "task families" (e.g., weak data, coarse solvers, missing physics).
- 2,511 episodes generated by evaluating 9 different models (including GPT-6 Astra, the Claude 5.5 family, and open-weight models like Qwen and Gemma).
- Exact ground truth: Because the "real" sample is itself a high-fidelity simulation, the true physical properties are known perfectly for scoring.
Key Insights and Ideas
- A Good Fit is a Trap: Every model tested could produce "excellent-looking" fits. Without scientific judgment, they reported these fits as high-confidence truths, even when the underlying parameters were off by 10-20%.
- Diagnosis $\neq$ Correction: Interestingly, many models (like Claude 5.5 Opus) could correctly name the missing physics (e.g., "there is likely a contact resistance") but still failed to provide accurate intervals. Diagnosis is a linguistic skill; correction is an experimental one.
- The "Gentle Heating" Strategy: The most successful agents (Astra and Fable 5.2) didn't just complain about the simulator; they worked around it. By running low-power, short-duration experiments, they minimized the effects of missing physics (like temperature-dependent $k$), making the simple simulator "effectively" correct.
- Reasoning Effort vs. Latent Skill: Claude 5.5 models failed at default settings but reached parity with Astra when forced to use "maximum reasoning" (writing 13–31x more tokens). This suggests the capability is latent but not prioritized at lower "effort" levels.
Key Results
- Calibration Gap: At default settings, most models' 95% intervals contained the truth only 20%–62% of the time. In contrast, GPT-6 Astra (94%) and Claude Fable 5.2 (99%) remained highly calibrated.
- The Cost of Thinking: Claude 5.5 Opus improved its interval score more than tenfold at maximum reasoning, but at a massive token cost.
- Open-Weight Struggles: Models like Qwen3.8-27B and Gemma 4 31B showed almost no improvement with increased reasoning effort, remaining confidently wrong on physics-trap tasks.
Limitations and Future Work
- Domain Specificity: The current benchmark is limited to thermal physics. The authors invite contributions to extend this to other domains like chemistry or fluid dynamics.
- Human Baselines: The study lacks a direct comparison to human scientists, though a "play-it-yourself" interface has been released to collect this data.
- Fixed Budgets: The agent is limited to four experiments; it's unclear if a larger budget would allow weaker models to "stumble" into the correct strategy.
The authors have released the environment, graders, and harness to encourage the development of agents that don't just fit data, but actually judge it.
Summary updated