TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real Labs

Yang Wan, Linchao ZhuarXiv 2026 · arXiv

TL;DR

TwinLab-Bench is a new benchmark designed to evaluate AI agents on their ability to design and execute original scientific experiments. By using "digital twins" grounded in real-world laboratory records, the benchmark reveals that even frontier models like Claude Fable 5.2 struggle with the nuances of experimental design, scoring an average of only 45.0 out of 100.

What Problem Does This Paper Address?

While AI has become adept at analyzing existing datasets or recalling scientific facts, the "holy grail" of an AI scientist is the ability to design its own experiments. Existing benchmarks typically evaluate agents in "closed loops"—asking them to pick from a predefined list of options or perform standardized data analysis.

The challenge is that evaluating open-ended experimental design requires a real laboratory, which is prohibitively slow and expensive for large-scale benchmarking. Conversely, simple simulators often lack the "messiness" and grounding of real-world physics. TwinLab-Bench bridges this gap by creating "agentic twins" of real labs, allowing AI agents to propose any experiment they want, with results generated from actual historical lab records and deterministic calculators.

The benchmark categorizes scientific tasks across three major divisions: Engineering, Physical Sciences, and Life Sciences.

Methods

The authors introduce a multi-agent architecture to simulate the scientific process:

  1. Actor Agent: The AI being tested. It receives a "handoff" (initial lab notes and protocols) and must propose hypotheses and design experiments within a set budget of time and materials.
  2. TwinLab Agent: This agent serves as the environment. It doesn't just "guess" results; it uses a suite of tools to read raw laboratory files and run "calculators"—deterministic programs built from real data that can interpolate or locally extrapolate results for experiments the original lab might not have performed exactly.
  3. Judge Agent: A vision-language model that scores the Actor's final report against a "hidden truth" (the actual eventual findings of the real-world lab).
The benchmark evaluates the most difficult "open" ability: designing original experiments within a constrained budget.

This setup ensures that the environment is "grounded." Without this grounding, the authors found that standard LLMs hallucinate numbers that deviate from real records 41% of the time.

Datasets

The benchmark consists of 52 tasks derived from real research records, including open lab notebooks, raw instrument exports, and published papers. These tasks are divided into:

  • Physical Sciences (20 tasks): Physics, Earth Sciences, and Chemistry.
  • Life Sciences (20 tasks): Marine Biology, Medicine, and Molecular/Cell Biology.
  • Engineering (12 tasks): Chemical, Electrical, Mechanical, and Materials Engineering.

The benchmark supports over 100 experimental action types, ranging from "extracting RNA" to "imaging broken steel bars with an SEM."

Key Insights and Ideas

  • The "Knowing-Doing Gap": One of the most interesting findings is that models often write a perfect experimental plan but then fail to follow it or fail to correctly interpret the data they receive. In several cases, models stated a decision in full before the run but still lost points because they couldn't execute the logic during the "wet lab" simulation.
  • The "Discriminating Test" Failure: The most common reason for failure wasn't a lack of knowledge, but poor scientific judgment. Agents frequently "accepted a mechanism without a discriminating test"—essentially jumping to a conclusion because the data looked plausible, rather than running a control experiment to rule out alternatives.
  • Grounding is Essential: The "TwinLab" approach almost eliminates the inconsistent and hallucinated readings common when using a raw LLM as a simulator. This makes the benchmark a much more rigorous test of scientific reasoning rather than just a test of "vibe-based" prediction.

Key Results

The authors evaluated 13 frontier models. The results show that autonomous science remains a high bar:

  • Top Performer: Claude Fable 5.2 achieved the highest average score of 45.0.
  • Runner Up: Claude Opus 5.5 followed with 40.2.
  • Cost vs. Performance: There is a clear correlation between the cost of the agent (reasoning tokens/compute) and the final score, but even the most expensive runs failed to clear the 50% mark on average.
A comparison of mean scores versus the cost of the actor agent shows that while more compute helps, a significant performance gap remains.

Limitations and Future Work

While TwinLab-Bench is a significant step forward, it has limitations:

  • Simulated Physicality: The benchmark focuses on experimental decisions, not the physical dexterity required to operate lab equipment.
  • Local Extrapolation: The "calculators" are only valid near the conditions recorded in the original lab data. If an agent designs an experiment wildly outside the scope of the original research, the system may hit a "simulation limit."
  • Future Directions: The authors suggest expanding the benchmark to include more complex, multi-stage research campaigns and exploring how agents can better handle "scarce materials" that cannot be recovered once used in a failed experiment.

Summary updated

Abstract

AI agents are increasingly expected to act as AI scientists, which requires three increasingly open abilities: carrying out a given computational workflow correctly, choosing the next measurements from a given design space, and designing their own experiments. Existing benchmarks mainly evaluate the first two abilities, in a terminal or in a closed loop that selects from predefined candidates. Evaluating the third ability requires reliably executing an experiment that an agent designs, with readings grounded in a real laboratory. We introduce TwinLab-Bench, which evaluates AI scientists on self-designed experiments in 52 tasks from laboratories in the physical sciences, the life sciences, and engineering. The benchmark supports more than 100 experimental action types, and the agent's report is scored by comparison with a hidden answer taken from real laboratory records. Each task has its own TwinLab agent, an agent that serves as the environment and executes the experiments that the tested agent designs, reading the raw materials of the laboratory and calling calculators that are built from its records. Compared with a language model that sees only the task description, the TwinLab agent almost eliminates readings that deviate from the real records and inconsistent answers to a repeated request, and closed-book probes control for answers recalled from memory. We find that these tasks remain hard for frontier models: the strongest, Claude Fable 5.2, averages a score of only 45.0, and we conduct an empirical study of where it fails.
BibTeX
@misc{wan2026twinlab,
  author = {Yang Wan and Linchao Zhu},
  title = {{TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real Labs}},
  year = {2026},
  eprint = {2611.26248},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26248},
}
Figure