This paper introduces SciWorkBench, a new benchmark designed to move AI evaluation away from "toy" problems and toward the messy, iterative reality of laboratory research. While large language models (LLMs) are increasingly deployed as autonomous agents, the authors find that even the most advanced models struggle when asked to act like real scientists.
TL;DR
SciWorkBench evaluates AI agents on multi-stage, realistic scientific workflows across biology, chemistry, and physics. The results show that current agents are far from autonomous; even the best-performing systems achieve a mean score of only 57.38%, frequently failing at scientific self-correction and multi-modal evidence integration.
What Problem Does This Paper Address?
Most existing AI benchmarks for science are "static"—they test whether a model can answer a multiple-choice question or generate a snippet of code from a cleaned dataset. However, real science is rarely that tidy. It is a hypothesis-driven process where researchers must deal with noisy data, decide which tools to use, interpret ambiguous plots, and revise their conclusions when experiments fail.
The authors argue that there is a "missing piece" in current evaluations: workflow-level scientific judgment. An agent might be able to write a Python script, but can it decide if the resulting graph actually supports its initial hypothesis? SciWorkBench addresses this gap by providing agents with raw, often imperfect data and requiring them to navigate a full research scenario.
Methods
The researchers developed a unified scientific analysis framework that breaks down a research project into several stages: hypothesis formulation, raw data inspection, curation, preliminary and in-depth analysis, modeling, and final conclusion/revision.
To build the benchmark, PhD-level experts identified tasks from peer-reviewed literature and private research projects. These tasks were then implemented in dockerized environments, ensuring the agents had the necessary libraries but no access to the internet or reference solutions.
Evaluation is handled via a rubric-based "Agent Judge." Because scientific tasks often have multiple valid solution paths, exact-match grading is insufficient. Instead, a high-performing model (GPT-5.6 Sol) uses expert-authored rubrics to grade the agent’s reasoning, evidence usage, and final output.
Datasets
SciWorkBench consists of 98 high-quality tasks filtered from an initial pool of 647 candidates. The distribution covers:
- Biology (44.9%): Including genomics, molecular biology, and neuroscience.
- Chemistry & Materials (38.8%): Covering catalysis, organic chemistry, and nanoscale materials.
- Physics (16.3%): Spanning optical, particle, and condensed-matter physics.
Tasks range from "Raw data to Scientific conclusion" (the most common, at 55.7%) to complex iterative loops involving modeling and prediction.
Key Insights and Ideas
- The "Harness" Matters: Performance isn't just a property of the model (e.g., GPT-5 or Opus 5); it depends heavily on the "harness" (the system that prompts and coordinates the agent). Interestingly, the same harness can improve one model while degrading another.
- Knowledge vs. Procedure: Agents often "know" the relevant science but fail to translate that knowledge into a task-specific procedure. They might recognize a chemical concept but apply a plausible-sounding yet incorrect workflow to the specific data provided.
- The Multi-modal Bottleneck: Real science requires integrating text, numerical tables, and visual plots. Agents struggle to assign the right "role" to different modalities at different stages, often ignoring visual cues that a human scientist would spot instantly.
- Failure of Self-Correction: When agents "self-check," they usually only check if their code runs or if the file format is correct. They rarely check if the result is scientifically plausible given the evidence.
Key Results
The authors evaluated 13 different model-harness configurations. The results indicate a significant ceiling for current technology:
- Top Performer: The Codex harness paired with GPT-5.6 Sol achieved the highest mean score of 57.38.
- Close Second: Claude Code with Opus 5 followed closely at 57.31.
- Discipline Variance: No single agent dominated all fields. GPT-5.6 Sol led in Biology, while Opus 5 was strongest in Chemistry/Materials, and Qwen3.8-Max led in Physics.
- Low Floor: Some frontier models scored as low as 22.12, highlighting that scientific workflow completion is a much harder bar than standard reasoning tasks.
Limitations and Future Work
The authors acknowledge that SciWorkBench, while diverse, cannot cover the entire breadth of modern science. Additionally, the high cost of running long-horizon agents means each task was only run a limited number of times, which may introduce some variability in scores.
Future work will likely focus on developing better scaffolds—the "skeletons" that help agents keep track of intermediate evidence and perform more rigorous scientific backtracking when results conflict with domain constraints. The goal is to move from agents that can merely "execute" to agents that can truly "reason" through a discovery process.
Summary updated