TL;DR
TwinLab-Bench is a new benchmark designed to evaluate AI agents on their ability to design and execute original scientific experiments. By using "digital twins" grounded in real-world laboratory records, the benchmark reveals that even frontier models like Claude Fable 5.2 struggle with the nuances of experimental design, scoring an average of only 45.0 out of 100.
What Problem Does This Paper Address?
While AI has become adept at analyzing existing datasets or recalling scientific facts, the "holy grail" of an AI scientist is the ability to design its own experiments. Existing benchmarks typically evaluate agents in "closed loops"—asking them to pick from a predefined list of options or perform standardized data analysis.
The challenge is that evaluating open-ended experimental design requires a real laboratory, which is prohibitively slow and expensive for large-scale benchmarking. Conversely, simple simulators often lack the "messiness" and grounding of real-world physics. TwinLab-Bench bridges this gap by creating "agentic twins" of real labs, allowing AI agents to propose any experiment they want, with results generated from actual historical lab records and deterministic calculators.
Methods
The authors introduce a multi-agent architecture to simulate the scientific process:
- Actor Agent: The AI being tested. It receives a "handoff" (initial lab notes and protocols) and must propose hypotheses and design experiments within a set budget of time and materials.
- TwinLab Agent: This agent serves as the environment. It doesn't just "guess" results; it uses a suite of tools to read raw laboratory files and run "calculators"—deterministic programs built from real data that can interpolate or locally extrapolate results for experiments the original lab might not have performed exactly.
- Judge Agent: A vision-language model that scores the Actor's final report against a "hidden truth" (the actual eventual findings of the real-world lab).
This setup ensures that the environment is "grounded." Without this grounding, the authors found that standard LLMs hallucinate numbers that deviate from real records 41% of the time.
Datasets
The benchmark consists of 52 tasks derived from real research records, including open lab notebooks, raw instrument exports, and published papers. These tasks are divided into:
- Physical Sciences (20 tasks): Physics, Earth Sciences, and Chemistry.
- Life Sciences (20 tasks): Marine Biology, Medicine, and Molecular/Cell Biology.
- Engineering (12 tasks): Chemical, Electrical, Mechanical, and Materials Engineering.
The benchmark supports over 100 experimental action types, ranging from "extracting RNA" to "imaging broken steel bars with an SEM."
Key Insights and Ideas
- The "Knowing-Doing Gap": One of the most interesting findings is that models often write a perfect experimental plan but then fail to follow it or fail to correctly interpret the data they receive. In several cases, models stated a decision in full before the run but still lost points because they couldn't execute the logic during the "wet lab" simulation.
- The "Discriminating Test" Failure: The most common reason for failure wasn't a lack of knowledge, but poor scientific judgment. Agents frequently "accepted a mechanism without a discriminating test"—essentially jumping to a conclusion because the data looked plausible, rather than running a control experiment to rule out alternatives.
- Grounding is Essential: The "TwinLab" approach almost eliminates the inconsistent and hallucinated readings common when using a raw LLM as a simulator. This makes the benchmark a much more rigorous test of scientific reasoning rather than just a test of "vibe-based" prediction.
Key Results
The authors evaluated 13 frontier models. The results show that autonomous science remains a high bar:
- Top Performer: Claude Fable 5.2 achieved the highest average score of 45.0.
- Runner Up: Claude Opus 5.5 followed with 40.2.
- Cost vs. Performance: There is a clear correlation between the cost of the agent (reasoning tokens/compute) and the final score, but even the most expensive runs failed to clear the 50% mark on average.
Limitations and Future Work
While TwinLab-Bench is a significant step forward, it has limitations:
- Simulated Physicality: The benchmark focuses on experimental decisions, not the physical dexterity required to operate lab equipment.
- Local Extrapolation: The "calculators" are only valid near the conditions recorded in the original lab data. If an agent designs an experiment wildly outside the scope of the original research, the system may hit a "simulation limit."
- Future Directions: The authors suggest expanding the benchmark to include more complex, multi-stage research campaigns and exploring how agents can better handle "scarce materials" that cannot be recovered once used in a failed experiment.
Summary updated