SciWorkBench: A Benchmark for Real Scientific Workflows

Yuchen Zhang, Taolin Han, Jinghang Wang, Jinxin Wang, Yun Wu, Wai Yuet Chiu, Xiaogang Li, Yuting Fu, Kewei Sun, Jiaxin Li, Qile Jin, Jiajia Li, Ruodan Chen, Qingteng Chen, Jialu Nie, Junchao Li, Yifei Zhang, Chen Zhao, Hu Wei, Lin Qu, Yanni Yin, Bing ZhaoarXiv 2026 · arXiv

This paper introduces SciWorkBench, a new benchmark designed to move AI evaluation away from "toy" problems and toward the messy, iterative reality of laboratory research. While large language models (LLMs) are increasingly deployed as autonomous agents, the authors find that even the most advanced models struggle when asked to act like real scientists.

TL;DR

SciWorkBench evaluates AI agents on multi-stage, realistic scientific workflows across biology, chemistry, and physics. The results show that current agents are far from autonomous; even the best-performing systems achieve a mean score of only 57.38%, frequently failing at scientific self-correction and multi-modal evidence integration.

What Problem Does This Paper Address?

Most existing AI benchmarks for science are "static"—they test whether a model can answer a multiple-choice question or generate a snippet of code from a cleaned dataset. However, real science is rarely that tidy. It is a hypothesis-driven process where researchers must deal with noisy data, decide which tools to use, interpret ambiguous plots, and revise their conclusions when experiments fail.

The authors argue that there is a "missing piece" in current evaluations: workflow-level scientific judgment. An agent might be able to write a Python script, but can it decide if the resulting graph actually supports its initial hypothesis? SciWorkBench addresses this gap by providing agents with raw, often imperfect data and requiring them to navigate a full research scenario.

Figure 1. Comparison between existing benchmark framework and our real laboratory analysis framework for scientific-agent evaluation. (a) Existing benchmarks often simplify scientific tasks to cleaned inputs, predefined procedures, and final answers. (b) Our framework organizes scientific-agent tasks around real experimental workflows.

Methods

The researchers developed a unified scientific analysis framework that breaks down a research project into several stages: hypothesis formulation, raw data inspection, curation, preliminary and in-depth analysis, modeling, and final conclusion/revision.

To build the benchmark, PhD-level experts identified tasks from peer-reviewed literature and private research projects. These tasks were then implemented in dockerized environments, ensuring the agents had the necessary libraries but no access to the internet or reference solutions.

Evaluation is handled via a rubric-based "Agent Judge." Because scientific tasks often have multiple valid solution paths, exact-match grading is insufficient. Instead, a high-performing model (GPT-5.6 Sol) uses expert-authored rubrics to grade the agent’s reasoning, evidence usage, and final output.

Figure 2. Construction pipeline of SciWorkBench. Tasks are collected from real laboratory scenario, then implemented under dockerized environments, graded with human-provided rubrics, double-checked by both automated workflow and human expert before accepted and released.

Datasets

SciWorkBench consists of 98 high-quality tasks filtered from an initial pool of 647 candidates. The distribution covers:

  • Biology (44.9%): Including genomics, molecular biology, and neuroscience.
  • Chemistry & Materials (38.8%): Covering catalysis, organic chemistry, and nanoscale materials.
  • Physics (16.3%): Spanning optical, particle, and condensed-matter physics.

Tasks range from "Raw data to Scientific conclusion" (the most common, at 55.7%) to complex iterative loops involving modeling and prediction.

Figure 4. Distribution of SciWorkBench tasks by covered workflow stages.

Key Insights and Ideas

  • The "Harness" Matters: Performance isn't just a property of the model (e.g., GPT-5 or Opus 5); it depends heavily on the "harness" (the system that prompts and coordinates the agent). Interestingly, the same harness can improve one model while degrading another.
  • Knowledge vs. Procedure: Agents often "know" the relevant science but fail to translate that knowledge into a task-specific procedure. They might recognize a chemical concept but apply a plausible-sounding yet incorrect workflow to the specific data provided.
  • The Multi-modal Bottleneck: Real science requires integrating text, numerical tables, and visual plots. Agents struggle to assign the right "role" to different modalities at different stages, often ignoring visual cues that a human scientist would spot instantly.
  • Failure of Self-Correction: When agents "self-check," they usually only check if their code runs or if the file format is correct. They rarely check if the result is scientifically plausible given the evidence.

Key Results

The authors evaluated 13 different model-harness configurations. The results indicate a significant ceiling for current technology:

  • Top Performer: The Codex harness paired with GPT-5.6 Sol achieved the highest mean score of 57.38.
  • Close Second: Claude Code with Opus 5 followed closely at 57.31.
  • Discipline Variance: No single agent dominated all fields. GPT-5.6 Sol led in Biology, while Opus 5 was strongest in Chemistry/Materials, and Qwen3.8-Max led in Physics.
  • Low Floor: Some frontier models scored as low as 22.12, highlighting that scientific workflow completion is a much harder bar than standard reasoning tasks.

Limitations and Future Work

The authors acknowledge that SciWorkBench, while diverse, cannot cover the entire breadth of modern science. Additionally, the high cost of running long-horizon agents means each task was only run a limited number of times, which may introduce some variability in scores.

Future work will likely focus on developing better scaffolds—the "skeletons" that help agents keep track of intermediate evidence and perform more rigorous scientific backtracking when results conflict with domain constraints. The goal is to move from agents that can merely "execute" to agents that can truly "reason" through a discovery process.

Summary updated

Abstract

Language models are increasingly being used as agents capable of planning and executing projects, as well as refining their actions through feedback. This shift raises a question: can current agents complete projects like real scientists? To examine this question, we introduce SciWorkBench, a benchmark for evaluating agents' abilities on real science tasks across biology, chemistry, materials science, and physics. SciWorkBench is built from peer-reviewed literature or expert-contributed research scenarios. We also propose a unified scientific analysis framework validated by domain experts, which can categorize most tasks in SciWorkBench. Each task covers one or more stages of this framework. We evaluate up-to-date models within the Claude Code and Codex agent frameworks under a unified execution and grading protocol. The best-performing agent achieves a mean score of 57.38, indicating that current agents still struggle with realistic scientific workflows. Our failure analysis identifies three recurring weaknesses at the workflow level: difficulty translating scientific knowledge into task-specific procedures, weak integration of evidence across different modalities, and limited capacity for scientific self-correction. These findings highlight the importance of workflow-based evaluation for measuring progress toward reliable AI systems for scientific research.
BibTeX
@misc{zhang2026sciworkbench,
  author = {Yuchen Zhang and Taolin Han and Jinghang Wang and Jinxin Wang and Yun Wu and Wai Yuet Chiu and Xiaogang Li and Yuting Fu and Kewei Sun and Jiaxin Li and Qile Jin and Jiajia Li and Ruodan Chen and Qingteng Chen and Jialu Nie and Junchao Li and Yifei Zhang and Chen Zhao and Hu Wei and Lin Qu and Yanni Yin and Bing Zhao},
  title = {{SciWorkBench: A Benchmark for Real Scientific Workflows}},
  year = {2026},
  eprint = {2611.22490},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22490},
}
Figure