Towards Multimodal Data-Driven Scientific Discovery Powered by LLM Agents

Fan Liu, Xiaozhao Zeng, Hao LiuICLR 2026 · OpenReview

Executive Summary

The researchers introduce MoSciBench, the first benchmark designed to evaluate Large Language Model (LLM) agents on multimodal, repository-level scientific discovery. Their findings reveal that while LLMs are becoming adept at coding and basic data manipulation, the "final boss" of automated science is cross-modal alignment—the ability to correctly link and integrate heterogeneous data types like satellite imagery, genotype matrices, and time series.

The Problem: Beyond Unimodal Simplifications

Current benchmarks for AI in science (like ScienceAgentBench or DiscoveryBench) often suffer from two major simplifications: they are unimodal (focusing on a single data type like a CSV table) and slice-level (handling small, isolated fragments of data).

Comparison of unimodal slice-level tasks versus multimodal repository-level tasks

In the real world, scientific discovery is messy and integrative. A climate scientist doesn't just look at a spreadsheet; they must align satellite images with temporal storm tracks and ground-station metadata. This paper addresses the lack of a rigorous evaluation framework for "repository-level" discovery, where an agent must navigate a complex file system, align disparate data sources, and execute an end-to-end hypothesis verification workflow.

Methods: Constructing MoSciBench

To bridge this gap, the authors developed a principled four-stage pipeline to construct MoSciBench:

The four-stage pipeline for constructing MoSciBench
  1. Raw Data Extraction: Sourcing datasets and research questions directly from peer-reviewed studies.
  2. Multimodal Processing & Alignment: Cleaning and standardizing data so that different modalities (e.g., linking a subject's ID in a table to their specific physiological time series) are computable.
  3. Task Formulation: Translating research questions into executable tasks with verifiable "gold" answers.
  4. Human Verification: Ensuring every task is solvable and the ground truth is 100% consistent with the provided data.

To evaluate agents, the researchers tested four frameworks—ReAct (reasoning + acting), DataVoyager (modular planning), Reflexion (iterative self-correction), and SelfDebug—across various LLM families including GPT-4o-mini, DeepSeek-V3, and Qwen.

Datasets and Scope

The core contribution, MoSciBench, consists of 88 end-to-end tasks spanning:

  • Six Domains: Climate Science, Biomedical Engineering, Cheminformatics, Health Psychology, Population Genomics, and Earth Science.
  • Seven Modalities: Multi-sensor time series, tabular data, satellite imagery, mass spectra, molecular structures, genotype matrices, and HDF matrices.
  • Five Discovery Categories: Descriptive analysis, correlation testing, causal inference, predictive modeling, and pattern discovery.

Key Insights and Results

  • Alignment is the Dominant Bottleneck: Over 60% of agent failures were due to cross-modal alignment issues. Agents struggle to "join" data correctly—for example, failing to match a geographic grid in an image to a specific timestamp in a weather log.
  • Low Ceiling for Current Agents: Even the strongest configuration achieved only 48.94% accuracy, highlighting that multimodal discovery is substantially harder than unimodal tasks.
  • Data Grounding is Essential: A "NoDataGuess" baseline (asking the LLM to guess the answer based on its internal knowledge without looking at the files) collapsed to near-zero accuracy. This proves that MoSciBench successfully tests data-driven reasoning rather than just memorized facts.
  • Process Over Raw Power: "Workflow scaffolding"—breaking a complex task into explicit sub-steps like data loading, alignment, and then modeling—consistently boosted accuracy by 5.7% and reduced alignment errors by 5–10%. This suggests that for complex science, how an agent thinks is as important as the model's size.
  • Domain Knowledge is a Double-Edged Sword: Surprisingly, providing agents with specific domain knowledge snippets (definitions, formulas) often decreased performance (a 3.5% drop on average). The authors suggest this "naïve injection" introduces noise or leads the agent to overthink, causing misalignment with the actual data provided.
  • Diminishing Returns on "Thinking": While increasing inference-time computation (using "Best-of-N" strategies) helped initially, performance peaked at N=3 and then declined. More "thinking" often led to the accumulation of low-quality, erroneous generations that eventually outweighed the correct ones.

Limitations and Future Work

The authors acknowledge that current agents remain "fragile" when faced with high-dimensional or noisy modalities like genotype matrices. While workflow scaffolding helps, it is currently a "lightweight" manual intervention.

Future work should focus on:

  • Adaptive Scaffolding: Developing agents that can autonomously decompose complex multimodal tasks without human-defined templates.
  • Robust Alignment Modules: Creating specialized tools or prompting strategies specifically designed to handle the "join" operations between different data formats.
  • Efficiency: Reducing the high API costs associated with iterative agents to make automated discovery more viable for large-scale research.

Summary updated

Abstract

Recent advances in large language models (LLMs) have enabled agents that automate scientific discovery by interpreting data, generating analysis pipelines, and executing them with computational tools. However, existing benchmarks remain largely limited to unimodal datasets and slice-level tasks, overlooking the fact that real discovery requires multimodal integration, modeling, and hypothesis-driven reasoning. To address this gap, we introduce MoSciBench, the first benchmark for multimodal scientific discovery, constructed from peer-reviewed studies through a principled four-stage pipeline. MoSciBench spans six scientific domains, seven data modalities, and five categories of discovery questions, yielding 88 individual, end-to-end, data-driven tasks. Each task is designed as a cross-modal hypothesis verification workflow, requiring agents to align and integrate heterogeneous datasets before modeling and reasoning. We further evaluate four representative agent frameworks across multiple LLM families. Results show that multimodal discovery is substantially harder than unimodal tasks: even the strongest agents achieve only 48.94\% accuracy, with over 60\% of failures due to cross-modal alignment. Lightweight workflow scaffolding consistently improves performance, reducing alignment errors by 5–10\% and raising accuracy by 5.7\% on average. Our benchmark and evaluation framework thus establish a rigorous testbed for advancing LLM agents toward realistic, multimodal scientific discovery.
BibTeX
@inproceedings{liu2026towards,
  author = {Fan Liu and Xiaozhao Zeng and Hao Liu},
  title = {{Towards Multimodal Data-Driven Scientific Discovery Powered by LLM Agents}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=kZHSvETWdi},
}
Figure