Executive Summary
The researchers introduce MoSciBench, the first benchmark designed to evaluate Large Language Model (LLM) agents on multimodal, repository-level scientific discovery. Their findings reveal that while LLMs are becoming adept at coding and basic data manipulation, the "final boss" of automated science is cross-modal alignment—the ability to correctly link and integrate heterogeneous data types like satellite imagery, genotype matrices, and time series.
The Problem: Beyond Unimodal Simplifications
Current benchmarks for AI in science (like ScienceAgentBench or DiscoveryBench) often suffer from two major simplifications: they are unimodal (focusing on a single data type like a CSV table) and slice-level (handling small, isolated fragments of data).
In the real world, scientific discovery is messy and integrative. A climate scientist doesn't just look at a spreadsheet; they must align satellite images with temporal storm tracks and ground-station metadata. This paper addresses the lack of a rigorous evaluation framework for "repository-level" discovery, where an agent must navigate a complex file system, align disparate data sources, and execute an end-to-end hypothesis verification workflow.
Methods: Constructing MoSciBench
To bridge this gap, the authors developed a principled four-stage pipeline to construct MoSciBench:
- Raw Data Extraction: Sourcing datasets and research questions directly from peer-reviewed studies.
- Multimodal Processing & Alignment: Cleaning and standardizing data so that different modalities (e.g., linking a subject's ID in a table to their specific physiological time series) are computable.
- Task Formulation: Translating research questions into executable tasks with verifiable "gold" answers.
- Human Verification: Ensuring every task is solvable and the ground truth is 100% consistent with the provided data.
To evaluate agents, the researchers tested four frameworks—ReAct (reasoning + acting), DataVoyager (modular planning), Reflexion (iterative self-correction), and SelfDebug—across various LLM families including GPT-4o-mini, DeepSeek-V3, and Qwen.
Datasets and Scope
The core contribution, MoSciBench, consists of 88 end-to-end tasks spanning:
- Six Domains: Climate Science, Biomedical Engineering, Cheminformatics, Health Psychology, Population Genomics, and Earth Science.
- Seven Modalities: Multi-sensor time series, tabular data, satellite imagery, mass spectra, molecular structures, genotype matrices, and HDF matrices.
- Five Discovery Categories: Descriptive analysis, correlation testing, causal inference, predictive modeling, and pattern discovery.
Key Insights and Results
- Alignment is the Dominant Bottleneck: Over 60% of agent failures were due to cross-modal alignment issues. Agents struggle to "join" data correctly—for example, failing to match a geographic grid in an image to a specific timestamp in a weather log.
- Low Ceiling for Current Agents: Even the strongest configuration achieved only 48.94% accuracy, highlighting that multimodal discovery is substantially harder than unimodal tasks.
- Data Grounding is Essential: A "NoDataGuess" baseline (asking the LLM to guess the answer based on its internal knowledge without looking at the files) collapsed to near-zero accuracy. This proves that MoSciBench successfully tests data-driven reasoning rather than just memorized facts.
- Process Over Raw Power: "Workflow scaffolding"—breaking a complex task into explicit sub-steps like data loading, alignment, and then modeling—consistently boosted accuracy by 5.7% and reduced alignment errors by 5–10%. This suggests that for complex science, how an agent thinks is as important as the model's size.
- Domain Knowledge is a Double-Edged Sword: Surprisingly, providing agents with specific domain knowledge snippets (definitions, formulas) often decreased performance (a 3.5% drop on average). The authors suggest this "naïve injection" introduces noise or leads the agent to overthink, causing misalignment with the actual data provided.
- Diminishing Returns on "Thinking": While increasing inference-time computation (using "Best-of-N" strategies) helped initially, performance peaked at N=3 and then declined. More "thinking" often led to the accumulation of low-quality, erroneous generations that eventually outweighed the correct ones.
Limitations and Future Work
The authors acknowledge that current agents remain "fragile" when faced with high-dimensional or noisy modalities like genotype matrices. While workflow scaffolding helps, it is currently a "lightweight" manual intervention.
Future work should focus on:
- Adaptive Scaffolding: Developing agents that can autonomously decompose complex multimodal tasks without human-defined templates.
- Robust Alignment Modules: Creating specialized tools or prompting strategies specifically designed to handle the "join" operations between different data formats.
- Efficiency: Reducing the high API costs associated with iterative agents to make automated discovery more viable for large-scale research.
Summary updated