BLADE: Benchmarking Language Model Agents for Data-Driven Science

Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A. Merrill, Jeffrey Heer, Tim Althoff

TL;DR

The paper introduces BLADE, a novel benchmark designed to evaluate language model (LM) agents on open-ended, data-driven scientific discovery tasks. Unlike traditional benchmarks that focus on single-answer code execution, BLADE uses a flexible, graph-based evaluation framework to match an agent's multi-step analytical decisions against a diverse corpus of expert-annotated ground truths. The evaluation reveals that while current LMs possess strong foundational knowledge, they struggle to generate diverse, complex statistical analyses unless paired with iterative data-interaction environments.

What Problem Does This Paper Address?

Answering scientific research questions from data is rarely a straight line. It requires what statisticians call navigating the "garden of forking paths"—making a series of nuanced, subjective decisions about which variables to control for, how to transform data, and which statistical models to apply.

While LM-based agents are increasingly capable of writing code, existing benchmarks fall short in evaluating them as true scientific partners. They typically focus on either simple, deterministic tasks (e.g., "calculate the mean of column X") or optimizing machine learning model accuracy. They fail to assess the intermediate planning and reasoning steps crucial for rigorous scientific inquiry. Furthermore, because there are often multiple valid ways to analyze the same dataset, evaluating open-ended agent responses automatically without penalizing valid alternative approaches has been an open challenge.

Methods

To bridge this gap, the authors developed BLADE, which evaluates agents on two primary tasks:

  1. Discerning Justifiable Decisions: A set of multiple-choice questions testing whether an agent can identify valid vs. invalid analytical choices given a research question and dataset.
  2. Generating an End-to-End Analysis: The agent must output a list of conceptual variables, an executable data transformation function, and a statistical modeling function.
Figure 1: Overview of BLADE. We gathered research questions and datasets from existing research papers, crowd-sourced analysis studies and statistic textbooks as well as analyses from expert annotators (boxes 1-2-3, and Sec. 3). Given a research question and dataset, LM agents generate a full analysis containing the relevant conceptual variables, a data transform function, and a statistical modeling function (boxes 1-4-5, and Sec. 4.2). BLADE automatically evaluates this against the ground truth (box 6 and Sec. 5).

To handle the open-ended nature of the generation task, BLADE introduces a flexible automatic evaluation pipeline:

  • Conceptual Variables: Uses an LM (GPT-4o) to evaluate the semantic equivalence of the agent's proposed variables against the ground truth, ensuring different phrasings of identical concepts are matched correctly.
  • Data Transformations: Converts the agent's Python code into a standardized column data flow graph. This allows BLADE to perform value matching and fuzzy graph isomorphism matching, rewarding partial credit for correct intermediate steps regardless of code structure or execution order.
  • Statistical Models: Translates the modeling code into natural language specifications to verify if the agent selected the correct model type and appropriate variables.
Figure 2: To allow flexible and fine-grained matching, we represent transforms in code (left) as a column data flow graph G (right). The nodes in blue are column indicator nodes P, and the nodes in orange are transform nodes T. Details of the data flow graph formalization are in Appendix A.3

Datasets

BLADE consists of 12 real-world datasets and research questions curated from peer-reviewed scientific literature and statistics textbooks. These cover diverse domains such as behavioral sciences, economics, education, and evolutionary biology (e.g., analyzing whether soccer players with darker skin tones receive more red cards, or if hurricane name femininity affects public preparedness).

The benchmark contains:

  • 188 multiple-choice questions for the discernment task.
  • 536 ground-truth analysis decisions compiled from independent analyses performed by 11 expert data analysts.

Key Insights and Ideas

  • The Diversity Deficit: LMs are highly capable of generating basic, error-free code, but they lack analytical diversity. When given an open-ended prompt, they tend to default to the simplest possible analysis, failing to explore alternative justifiable control variables or complex model architectures that human experts routinely consider.
  • Coding Benchmarks Don't Predict Scientific Acumen: The authors found that a model's performance on standard code generation benchmarks (like HumanEval) does not strongly correlate with its ability to explore diverse, valid scientific pathways on BLADE. Scientific data analysis is fundamentally a reasoning and planning problem, not just a syntax problem.
  • The Power of Iteration: Agents evaluated within an interactive environment (using the ReAct framework to write code, view execution outputs, and refine their strategy) consistently generated more diverse and comprehensive analyses than those operating in a single-turn, zero-shot setting.

Key Results

  • Decision Discernment: Top closed-source models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) and large open-source models (Llama3 70B) performed exceptionally well on the multiple-choice tasks, achieving 84% to 89% accuracy, significantly outperforming the 26% random chance baseline. Conversely, smaller code-specific models struggled.
Figure 3: Accuracy scores and 95% confidence intervals for different models on BLADE’s 188 MCQs (168 for transformations and 20 for conceptual variables).
  • End-to-End Generation: Models struggled heavily with complex statistical modeling. In one-turn settings, GPT-4o's coverage of the expert ground truth for forming statistical models with conceptual variables was below 13%, and its coverage for operationalizing variables was below 27%.
  • Agent Framework Boosts: Transitioning from a one-turn setting to an interactive ReAct agent setting provided a reliable performance lift. For example, GPT-3.5 Turbo's decision-weighted F1-score improved from 30.5 to 37.2 when allowed to iteratively interact with the data sandbox.

Limitations and Future Work

The authors note several boundaries to the current iteration of BLADE:

  • Interpretation is Omitted: BLADE evaluates the planning and execution of data analysis but does not test whether an agent can accurately interpret statistical outputs or synthesize them into valid scientific conclusions.
  • Single-Table Assumption: The benchmark assumes all data is flattened into a single table, bypassing the challenges of relational database joins.
  • LM-Evaluation Dependency: The evaluation pipeline relies on LMs for semantic matching. While validated to be 92%–97% accurate, it introduces a minor vector for evaluation error.

Future directions include expanding benchmarks to evaluate how agents interpret results and exploring how grounding agents with external knowledge bases (like ArXiv papers) can improve their scientific reasoning capabilities.

Summary updated

Abstract

Data-driven scientific discovery requires the iterative integration of scientific domain knowledge, statistical expertise, and an understanding of data semantics to make nuanced analytical decisions, e.g., about which variables, transformations, and statistical models to consider. LM-based agents equipped with planning, memory, and code execution capabilities have the potential to support data-driven science. However, evaluating agents on such open-ended tasks is challenging due to multiple valid approaches, partially correct steps, and different ways to express the same decisions. To address these challenges, we present BLADE, a benchmark to automatically evaluate agents' multifaceted approaches to open-ended research questions. BLADE consists of 12 datasets and research questions drawn from existing scientific literature, with ground truth collected from independent analyses by expert data scientists and researchers. To automatically evaluate agent responses, we developed corresponding computational methods to match different representations of analyses to this ground truth. Though language models possess considerable world knowledge, our evaluation shows that they are often limited to basic analyses. However, agents capable of interacting with the underlying data demonstrate improved, but still non-optimal, diversity in their analytical decision making. Our work enables the evaluation of agents for data-driven science and provides researchers deeper insights into agents' analysis approaches.
BibTeX
@inproceedings{gu2024blade,
  author = {Ken Gu and Ruoxi Shang and Ruien Jiang and Keying Kuang and Richard-John Lin and Donghe Lyu and Yue Mao and Youran Pan and Teng Wu and Jiaqian Yu and Yikun Zhang and Tianmai M. Zhang and Lanyi Zhu and Mike A. Merrill and Jeffrey Heer and Tim Althoff},
  title = {{BLADE: Benchmarking Language Model Agents for Data-Driven Science}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.815},
}
Figure