BLADE: Benchmarking Language Model Agents for Data-Driven Science

Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A. Merrill, Jeffrey Heer, Tim Althoff

Abstract

BLADE (Benchmarking Language Model Agents for Data-Driven Science) is a novel evaluation framework designed to assess the capacity of AI agents to perform complex, open-ended scientific data analysis. Unlike existing benchmarks that focus on single-step tasks or fixed outcomes, BLADE leverages expert-defined "ground truth decision spaces" to evaluate the nuances of variable selection, data transformation, and statistical modeling. Our findings indicate that while top-tier models like GPT-4o show promise, they often struggle with the depth and diversity of analysis required in rigorous scientific inquiry.

Introduction

In scientific research, data analysis is rarely a linear path with a single correct answer. Instead, researchers navigate a "multiverse" of justifiable analytical decisions—from choosing control variables to selecting appropriate statistical models. Existing benchmarks for Language Models (LMs) often simplify this process into single-step calculations or focus solely on predictive accuracy, failing to capture the iterative, knowledge-intensive nature of real-world science. BLADE fills this gap by testing an agent's ability to comprehend data semantics, integrate domain knowledge, and discern justifiable decisions.

Comparison of BLADE requirements against existing benchmarks, highlighting its focus on data semantics, domain knowledge, and justifiable decision-making.

Methodology

BLADE addresses the complexity of scientific analysis through a multi-stage evaluation process. First, 11 expert analysts established a "decision space" for 12 research questions, capturing multiple valid analytical paths. To evaluate agent performance against this ground truth, BLADE employs three novel matching techniques:

  1. Conceptual Variable Matching: Uses an LM to determine if an agent's proposed variables are semantically equivalent to expert-defined ones.
  2. Fuzzy Graph Isomorphism: Represents data transformations as a "column data flow graph," allowing the system to recognize logical steps even if the code structure varies.
  3. Statistical Model Matching: Combines semantic matching of model types with a check for the inclusion of correct conceptual variables in the final formula.
The BLADE benchmark workflow: from research questions and expert annotations to automated evaluation of agent-generated analysis against expert ground truth.

Datasets

The benchmark suite includes 12 real-world datasets spanning Behavioral Sciences, Evolutionary Biology, Finance, and Education. These datasets are paired with complex research questions, such as investigating the relationship between player skin tone and red cards in soccer, or the impact of gender on mortgage application approval rates. The suite includes 188 multiple-choice questions for decision-making and 536 ground-truth analysis decisions for end-to-end evaluation.

Results and Insights

Our evaluation reveals that top models like GPT-4o and Claude 3.5 Sonnet act as "basic" analysts; they are proficient at generating executable code (GPT-4o succeeded 96% of the time) but often miss the complex, nuanced approaches favored by human experts. For statistical modeling, LMs covered less than 13% of the justifiable approaches found in the expert ground truth.

Notably, general coding ability (e.g., performance on HumanEval) does not directly correlate with scientific rigor, which requires deep domain reasoning beyond writing bug-free code. However, agents using the ReAct framework—allowing for iterative thought, action in a sandbox, and observation—significantly outperformed one-turn models by refining their plans based on direct data interaction.

Conclusion

BLADE provides a rigorous framework for evaluating AI's potential as a scientific collaborator. While current models demonstrate strong coding foundations, significant gaps remain in their ability to explore the full breadth of justifiable scientific decisions. Future developments will focus on integrating external knowledge sources, such as academic papers, and improving the agent's ability to interpret final results and draw meaningful scientific conclusions.

Summary updated

Abstract

Data-driven scientific discovery requires the iterative integration of scientific domain knowledge, statistical expertise, and an understanding of data semantics to make nuanced analytical decisions, e.g., about which variables, transformations, and statistical models to consider. LM-based agents equipped with planning, memory, and code execution capabilities have the potential to support data-driven science. However, evaluating agents on such open-ended tasks is challenging due to multiple valid approaches, partially correct steps, and different ways to express the same decisions. To address these challenges, we present BLADE, a benchmark to automatically evaluate agents’ multifaceted approaches to open-ended research questions. BLADE consists of 12 datasets and research questions drawn from existing scientific literature, with ground truth collected from independent analyses by expert data scientists and researchers. To automatically evaluate agent responses, we developed corresponding computational methods to match different representations of analyses to this ground truth. Though language models possess considerable world knowledge, our evaluation shows that they are often limited to basic analyses. However, agents capable of interacting with the underlying data demonstrate improved, but still non-optimal, diversity in their analytical decision making. Our work enables the evaluation of agents for data-driven science and provides researchers deeper insights into agents’ analysis approaches.
BibTeX
@inproceedings{gu2024blade,
  author = {Ken Gu and Ruoxi Shang and Ruien Jiang and Keying Kuang and Richard-John Lin and Donghe Lyu and Yue Mao and Youran Pan and Teng Wu and Jiaqian Yu and Yikun Zhang and Tianmai M. Zhang and Lanyi Zhu and Mike A. Merrill and Jeffrey Heer and Tim Althoff},
  title = {{BLADE: Benchmarking Language Model Agents for Data-Driven Science}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.815},
}
Figure