BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories

Fei Lin, Tengchao Zhang, Ziyang Gong, Xiaotong Yu, Bohan Zhang, Yifan Zhou, Qihao Yang, Ji Dai, Dong Li, Yining Jiang, Qinghua Ni, Jun Huang, Qiang Zhang, Yue Zhou, Zhihang Zhong, Yonglin Tian, Fei-Yue Wang, Xue YangarXiv 2026 · arXiv

TL;DR

The researchers introduce BEAKER, a first-of-its-kind, expert-curated benchmark designed to evaluate Multimodal Large Language Models (MLLMs) as "embodied brains" for self-driving chemical laboratories. Their evaluation of 28 state-of-the-art models reveals a significant "execution gap": while models can follow high-level procedures, they struggle with the precise spatial grounding and fine-grained perception required to actually perform laboratory tasks.

What Problem Does This Paper Address?

Self-Driving Chemical Laboratories (SDCLs) aim to automate discovery by using robotic systems that can perceive, reason, and act autonomously. To function effectively, these systems need an "embodied brain"—a cognitive layer that translates visual inputs into physical actions under strict procedural and safety constraints.

However, existing benchmarks for MLLMs in chemistry focus primarily on high-level scientific knowledge (like predicting reaction outcomes) or basic safety monitoring. They fail to test embodied actionability: Can the model identify the exact point to grip a fallen vial? Can it plan a collision-free trajectory for a robotic arm? Can it recognize if a specific apparatus connection is secure? BEAKER fills this gap by shifting the focus from "passive scene understanding" to "execution-grounded reasoning."

Methods

The authors developed a rigorous, four-stage pipeline to create BEAKER, involving 10 chemistry experts and 12 embodied AI experts. Crucially, all 1,000 question-answer (QA) samples were manually authored and reviewed to avoid the self-bias inherent in AI-generated benchmarks.

Overview of BEAKER construction and task examples

The benchmark decomposes laboratory execution into four core capability dimensions:

  1. Actionability: Spatial tasks like affordance grounding (pointing to operable regions) and trajectory planning.
  2. Knowledge: Identifying apparatus, reagents, and recognizing safety norms.
  3. Experimentation: Understanding ongoing operations and apparatus setups.
  4. Reasoning: Inferring temporal changes and experimental outcomes from video.

The evaluation covers 11 distinct task types across both image and video modalities, using standardized prompts that require structured JSON outputs for objective scoring.

Datasets

BEAKER consists of 1,000 expert-curated QA samples derived from a pool of 883 images and 734 video clips. The visual data was sourced from 12 representative image repositories (e.g., HeinSight, Roboflow, LabPics) and 2 major video sources (IBM Lab Actions and JoVE Chemistry).

Unlike previous datasets, BEAKER specifically selects for "first-person" or "execution-related" viewpoints that mimic the visual input a robot would actually receive at a laboratory workbench.

Key Insights and Ideas

  • The Recognition-Grounding Gap: There is a massive disparity between a model's ability to name an object and its ability to interact with it. Models might correctly identify a "separatory funnel" in an MCQ but fail completely to locate the specific stopcock handle needed to operate it.
  • Domain Knowledge $\neq$ Embodied Skill: Surprisingly, science-oriented and chemistry-specialized MLLMs do not consistently outperform general-purpose models. This suggests that "book smarts" in chemistry don't automatically translate to "lab smarts" in execution.
  • Temporal Reasoning Bottlenecks: Models struggle most with mid-length videos (~50s). While they can often guess the "next step" in a very short clip, they fail to precisely segment and order steps in continuous procedures where boundaries are visually subtle.
Qualitative examples of successful and failed model predictions

Key Results

The researchers evaluated 28 models, including GPT-5.5, Claude 4.7, and specialized models like ChemVLM.

  • Human-Model Gap: Human experts achieved an overall score of 93.3, while the best-performing model (Claude Opus 4.7) reached only 59.2.
  • Actionability is the Bottleneck: Across all models, Actionability was the lowest-scoring dimension (averaging 22.2). In coordinate-based tasks (locating where to act), the mean Intersection over Union (IoU) for all models was less than 35%.
  • Visual Importance: Ablation studies (Figure 6) confirmed that visual information is critical; removing images caused performance to plummet toward random-choice baselines for most models.
Visual input ablation results across different models

Limitations and Future Work

The primary limitation is that BEAKER is an offline, single-turn benchmark. It evaluates perception and reasoning at a single moment but does not test closed-loop interaction, where a model must adapt to real-time feedback or recover from errors during an actual experiment.

Future work aims to integrate BEAKER with high-fidelity chemical simulation environments (like LabUtopia). This would allow for "agentic" evaluation, where the model's proposed trajectories and actions are actually executed in a virtual lab to see if the experiment succeeds, providing a truer measure of an MLLM's potential as a robotic brain.

Summary updated

Abstract

Self-Driving Chemical Laboratories (SDCLs) are moving chemical experimentation toward embodied automation, where machines must perceive laboratory scenes, reason about experimental states, and act under physical, procedural, and safety constraints. Although Multimodal Large Language Models (MLLMs) are increasingly considered as embodied brains for such systems, existing laboratory benchmarks mainly focus on safety, anomaly detection, or general scientific understanding, leaving their execution-oriented embodied capabilities underexplored. To fill this gap, we introduce BEAKER, a Benchmark for evaluating Embodied Actionability, Knowledge, Experimentation, and Reasoning in SDCLs. BEAKER contains 1,000 expert-curated question-answer samples across 11 image and video task types, covering key judgments required for chemical laboratory execution, including affordance grounding, trajectory planning, equipment, state, and safety recognition, apparatus setup, operation and step reasoning, and temporal procedure reasoning. All questions and annotations are manually created and reviewed by chemistry and embodied Artificial Intelligence (AI) experts, without AI-assisted generation or annotation. We evaluate 28 general-purpose, science-oriented, chemistry-specialized, and embodied MLLMs in an offline, single-turn setting. Results show that current models can follow coarse experimental procedures in videos, but still struggle to ground actions into operable regions and feasible trajectories, to recognize fine-grained laboratory apparatus, states, and safety norms, and to order steps in mid-length procedures. BEAKER thus reveals a clear gap between coarse procedural understanding and the precise, action-grounded perception required for laboratory execution, providing a diagnostic benchmark for future MLLMs in SDCLs.
BibTeX
@misc{lin2026beaker,
  author = {Fei Lin and Tengchao Zhang and Ziyang Gong and Xiaotong Yu and Bohan Zhang and Yifan Zhou and Qihao Yang and Ji Dai and Dong Li and Yining Jiang and Qinghua Ni and Jun Huang and Qiang Zhang and Yue Zhou and Zhihang Zhong and Yonglin Tian and Fei-Yue Wang and Xue Yang},
  title = {{BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories}},
  year = {2026},
  eprint = {2611.06911},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06911},
}
Figure