TL;DR
The paper introduces AblationBench, a new benchmark suite designed to evaluate how well language model agents can plan ablation experiments in AI research. By testing models from both an author's perspective (proposing experiments) and a reviewer's perspective (identifying missing experiments), the study reveals that even frontier LMs struggle with scientific design, capturing only about 45% of human-conducted ablations.
What Problem Does This Paper Address?
While AI "co-scientists" are rapidly advancing in tasks like writing code or generating hypotheses, evaluating their ability to make meaningful scientific contributions remains difficult. A cornerstone of empirical AI research is the ablation study—the practice of systematically removing or modifying components of a machine learning method to understand what actually drives performance.
Planning a good ablation study requires deep domain expertise to identify core components, propose feasible alternatives, and avoid trivial hyperparameter tuning. Until now, evaluating an AI's capacity for this type of high-level experimental design has relied on manual human grading. This paper addresses the lack of automated, standardized benchmarks for ablation planning from raw scientific text.
Methods
AblationBench splits the problem into two distinct tasks to simulate real-world research scenarios:
- AuthorAblation: Proposing ablation experiments given only a paper's title, abstract, and method section (truncating the actual experiments).
- ReviewerAblation: Acting as a peer reviewer to suggest critical missing ablations based on a full paper preprint.
To evaluate these tasks without relying on expensive human loops, the authors designed automated LM-based judges. These judges compare an AI planner's generated proposals against a "gold standard" ground truth (either the actual ablations the authors ran, or the missing ablations requested by real ICLR reviewers). To ensure fairness and eliminate biases, the judges use model ensembles, blind side-by-side matching, and randomized shuffling of the inputs.
The authors benchmarked two types of AI planners:
- LM-Planner: A standard LLM using a single Chain-of-Thought (CoT) prompt.
- Agent-Planner: A multi-step agent built on the SWE-agent framework, allowed to browse files and iteratively refine its plan.
Datasets
AblationBench introduces two primary evaluation datasets:
- AuthorAblation Dataset: Contains 83 machine learning conference papers spanning 14 distinct conferences (including ICML, NeurIPS, CVPR, and ACL). It features 230 human-annotated ablation experiments extracted directly from the published papers.
- ReviewerAblation Dataset: Comprises 350 ICLR submissions (from 2023–2025) paired with their official peer reviews. The dataset focuses specifically on cases where reviewers explicitly called out missing ablation experiments.
- JudgeEval: A separate, manually verified dataset used specifically to validate that the automated LM judges align well with human reasoning.
Key Insights and Ideas
- The Grounding Trade-Off: The authors discovered a fascinating inverse performance trend among models. Highly "grounded" models (like Qwen) excel at the author task because they strictly adhere to the provided method text. However, this strictness hurts them on the reviewer task, which requires creative extrapolation to see what isn't there. Conversely, less grounded models (like Llama) perform better as reviewers because they are more willing to propose abstract, missing ideas.
- Chain-of-Thought Beats Agents: Surprisingly, a single LLM call with Chain-of-Thought prompting (
LM-Planner) generally outperformed the more complex, multi-stepAgent-Plannerworkflow, while being significantly faster and cheaper. - Removal is Easier Than Modification: Across all models, AIs were much better at suggesting the complete removal of a component than proposing a clever modification or substitution. Proposing valid alternatives requires domain-specific intuition that current models lack.
Key Results
- Significant Human-AI Gap: On a subset of the AuthorAblation task, human PhD students achieved an F1 score of 0.65, whereas the best-performing frontier model (GPT-5.4) only achieved 0.42.
- Low Overall Recall: On average across both tasks, the best AI systems successfully identified only 45% of the original human-planned or reviewer-requested ablations.
- Reliable Automated Judges: The majority-vote
LMJudgeachieved a Cohen's kappa ($\kappa$) of 0.62 against human annotations, indicating a substantial and reliable agreement, validating its use as an automated evaluation metric.
Limitations and Future Work
A primary limitation acknowledged by the authors is that the benchmark evaluates models strictly against a fixed ground truth. If an AI planner proposes a highly creative, completely novel, and scientifically valid ablation experiment that the original authors simply didn't think of, AblationBench's automated judge will penalize it as a false positive.
Future iterations of the benchmark will need to explore methods for evaluating the intrinsic quality, feasibility, and novelty of an ablation plan, rather than relying solely on gold-label matching.
Summary updated