AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research

Talor Abramovich, Gal Chechik

TL;DR

The paper introduces AblationBench, a new benchmark suite designed to evaluate how well language model agents can plan ablation experiments in AI research. By testing models from both an author's perspective (proposing experiments) and a reviewer's perspective (identifying missing experiments), the study reveals that even frontier LMs struggle with scientific design, capturing only about 45% of human-conducted ablations.

What Problem Does This Paper Address?

While AI "co-scientists" are rapidly advancing in tasks like writing code or generating hypotheses, evaluating their ability to make meaningful scientific contributions remains difficult. A cornerstone of empirical AI research is the ablation study—the practice of systematically removing or modifying components of a machine learning method to understand what actually drives performance.

Planning a good ablation study requires deep domain expertise to identify core components, propose feasible alternatives, and avoid trivial hyperparameter tuning. Until now, evaluating an AI's capacity for this type of high-level experimental design has relied on manual human grading. This paper addresses the lack of automated, standardized benchmarks for ablation planning from raw scientific text.

Methods

AblationBench splits the problem into two distinct tasks to simulate real-world research scenarios:

  1. AuthorAblation: Proposing ablation experiments given only a paper's title, abstract, and method section (truncating the actual experiments).
  2. ReviewerAblation: Acting as a peer reviewer to suggest critical missing ablations based on a full paper preprint.
Figure 1. Overview of AblationBench. (Top) AuthorAblation is a collection of conference papers where the task of the planner is to generate ablation plan based on the paper’s method section. The judge evaluates the generated plan using manually extracted gold ablations from the paper. (Bottom) ReviewerAblation is a collection of ICLR submissions where the task of the planner is to generate missing ablation based on the full submission. The judge evaluates the generated ablations using the official reviews of the submission.

To evaluate these tasks without relying on expensive human loops, the authors designed automated LM-based judges. These judges compare an AI planner's generated proposals against a "gold standard" ground truth (either the actual ablations the authors ran, or the missing ablations requested by real ICLR reviewers). To ensure fairness and eliminate biases, the judges use model ensembles, blind side-by-side matching, and randomized shuffling of the inputs.

The authors benchmarked two types of AI planners:

  • LM-Planner: A standard LLM using a single Chain-of-Thought (CoT) prompt.
  • Agent-Planner: A multi-step agent built on the SWE-agent framework, allowed to browse files and iteratively refine its plan.

Datasets

AblationBench introduces two primary evaluation datasets:

  • AuthorAblation Dataset: Contains 83 machine learning conference papers spanning 14 distinct conferences (including ICML, NeurIPS, CVPR, and ACL). It features 230 human-annotated ablation experiments extracted directly from the published papers.
  • ReviewerAblation Dataset: Comprises 350 ICLR submissions (from 2023–2025) paired with their official peer reviews. The dataset focuses specifically on cases where reviewers explicitly called out missing ablation experiments.
  • JudgeEval: A separate, manually verified dataset used specifically to validate that the automated LM judges align well with human reasoning.

Key Insights and Ideas

  • The Grounding Trade-Off: The authors discovered a fascinating inverse performance trend among models. Highly "grounded" models (like Qwen) excel at the author task because they strictly adhere to the provided method text. However, this strictness hurts them on the reviewer task, which requires creative extrapolation to see what isn't there. Conversely, less grounded models (like Llama) perform better as reviewers because they are more willing to propose abstract, missing ideas.
  • Chain-of-Thought Beats Agents: Surprisingly, a single LLM call with Chain-of-Thought prompting (LM-Planner) generally outperformed the more complex, multi-step Agent-Planner workflow, while being significantly faster and cheaper.
  • Removal is Easier Than Modification: Across all models, AIs were much better at suggesting the complete removal of a component than proposing a clever modification or substitution. Proposing valid alternatives requires domain-specific intuition that current models lack.
Figure 3. Recall@5 performance for AuthorAblation of each planner approach across different LMs, separated by ablation type.

Key Results

  • Significant Human-AI Gap: On a subset of the AuthorAblation task, human PhD students achieved an F1 score of 0.65, whereas the best-performing frontier model (GPT-5.4) only achieved 0.42.
  • Low Overall Recall: On average across both tasks, the best AI systems successfully identified only 45% of the original human-planned or reviewer-requested ablations.
  • Reliable Automated Judges: The majority-vote LMJudge achieved a Cohen's kappa ($\kappa$) of 0.62 against human annotations, indicating a substantial and reliable agreement, validating its use as an automated evaluation metric.

Limitations and Future Work

A primary limitation acknowledged by the authors is that the benchmark evaluates models strictly against a fixed ground truth. If an AI planner proposes a highly creative, completely novel, and scientifically valid ablation experiment that the original authors simply didn't think of, AblationBench's automated judge will penalize it as a false positive.

Future iterations of the benchmark will need to explore methods for evaluating the intrinsic quality, feasibility, and novelty of an ablation plan, rather than relying solely on gold-label matching.

Summary updated

Abstract

Language model agents are increasingly used to automate scientific research, yet evaluating their scientific contributions remains a challenge. A key mechanism to obtain such insights is through ablation experiments. To this end, we introduce AblationBench, a benchmark suite for evaluating agents on ablation planning tasks in empirical AI research. It includes two tasks: AuthorAblation, which helps authors propose ablation experiments based on a method section and contains 83 instances, and ReviewerAblation, which helps reviewers find missing ablations in a full paper and contains 350 instances. For both tasks, we develop LM-based judges that serve as an automatic evaluation framework. Our experiments with frontier LMs show that these tasks remain challenging, with the best-performing LM system identifying only 45% of the original ablations on average, below human-level performance. We observe an inverse performance trend between the author and reviewer tasks, which we attribute to differences in model grounding. Lastly, we analyze the limitations of current LMs on these tasks, and find that chain-of-thought prompting outperforms an agent-based approach. Our data is available on https://huggingface.co/collections/ai-coscientist/ablationbench, and our code is available on https://github.com/ai-scientist-bench/ablation-bench .
BibTeX
@inproceedings{abramovich2026ablationbench,
  author = {Talor Abramovich and Gal Chechik},
  title = {{AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research}},
  booktitle = {AI4Science @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=mF0itTNVyG},
}
Figure