BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection

John Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy, Max Xu, Sam Schmidgall, Mark Malhotra, Shwetak Patel, Jimeng Sun, Daniel McDuff, Bo Dai, Xin LiuarXiv 2026 · arXiv

TL;DR

BuggyMLE is a new framework that synthetically scales the difficulty of machine learning engineering (MLE) tasks by injecting controlled "bugs" into real-world code repositories. By introducing the Code Complexity Index (CCI)—a structural metric that predicts task difficulty with remarkable accuracy ($R^2 = 0.951$)—the authors developed a curriculum that allowed a 9B-parameter model to rival frontier models like Claude 3.5 Opus and significantly improve its performance on general software engineering benchmarks.

What Problem Does This Paper Address?

As AI agents move from simple question-answering to autonomous research and engineering, we need better ways to evaluate them. Machine Learning Engineering (MLE) is a perfect "proving ground" because it requires navigating complex, multi-file codebases, running training pipelines, and optimizing measurable metrics.

However, current MLE benchmarks suffer from three major flaws:

  1. Oversimplification: They often focus on single-file edits, ignoring the "spaghetti" nature of real-world repositories.
  2. Saturation: Frontier models are already outperforming human reference solutions on existing suites.
  3. Curation Bottleneck: Manually creating high-quality, repository-level tasks with verified solutions is incredibly expensive and slow.

The authors ask: Can we automate the creation of difficult, realistic MLE tasks, and can we measure exactly how hard those tasks are?

Methods

The authors tackle this with a two-pronged approach: measuring difficulty and generating it.

1. The Code Complexity Index (CCI)

The researchers identified ten structural features of code—such as lines of code, dependency depth, and the number of "hunks" (blocks of modified lines) in a patch. They found that these structural properties alone, independent of the specific AI model being tested, are incredibly strong predictors of how likely a model is to solve a task.

2. The BuggyMLE Framework

To generate tasks, they built a four-stage pipeline:

  • Injector: An LLM agent explores a "clean" ML repository and injects breaking changes (bugs) across data loading, model architecture, or training code. The "difficulty" is controlled by a budget of how many files and lines the agent is allowed to break.
  • Oracle: An automated process that verifies which injected bugs actually degrade performance, creating a "ground truth" for what needs to be fixed.
  • Solver: The agent under test attempts to diagnose and repair the repository using a standard terminal environment.
  • Evaluator: A system that grades the repair based on metric recovery (e.g., does the model's accuracy return to baseline?) and a per-bug code review.
The BuggyMLE workflow for curating data and evaluating agents.

Datasets

The paper introduces BuggyMLEBench-Lite, a benchmark consisting of 88 oracle-verified tasks across four difficulty tiers (1, 5, 10, and 50 bugs per repository) derived from 22 diverse ML projects.

For training, they synthesized a massive dataset:

  • 6,674 bug-injected variants based on 222 source repositories.
  • ~260,000 lines of injected code.
  • Training Curriculum: They used this data to fine-tune a Qwen3.5-9B model using Supervised Fine-Tuning (SFT) and Reinforcement Learning (specifically Group Relative Policy Optimization, or GRPO).

Key Insights and Ideas

  • Structure Drives Difficulty: The most surprising finding is that structural features—especially patch size and complexity—explain 95% of the variance in solve rates across eight major public coding benchmarks. If a fix requires touching many files and hunks, it becomes exponentially harder for current models.
  • The "CCI Slider": By using CCI, the authors can precisely tune the difficulty of a synthetic task. As they "slide" the CCI up by injecting more bugs, solve rates for every model tested (from small open-weight models to GPT-4o) drop predictably.
  • MLE Training Generalizes to SWE: Training a model specifically to fix ML pipelines actually makes it a better general-purpose programmer. The researchers saw significant "transfer" to non-ML benchmarks like SWE-bench.
Examples of injected bugs, ranging from calibration errors to misconfigured label smoothing.

Key Results

  • Benchmark Scaling: On BuggyMLEBench-Lite, the top-performing model's solve rate plummeted from 95% on the easiest tier (1 bug) to just 9% on the hardest tier (50 bugs), proving the benchmark is far from saturated.
  • 9B Model vs. Flagships: Their fine-tuned Qwen3.5-9B model, trained on an "All Difficulties" curriculum, outperformed much larger models like Claude 3 Opus on the 10-bug tier.
  • Out-of-Distribution Transfer: The model's performance on SWE-bench Lite (general software engineering) jumped from 24% to 33%, and its LiveCodeBench (competitive programming) score rose from 69% to 71%.
CCI vs. Solve Rate: As complexity increases, the solve rate for all models follows a predictable downward trajectory.

Limitations and Future Work

  • Semantic vs. Structural Complexity: While CCI is a great predictor, it relies on structural features. It doesn't yet capture "semantic" difficulty—the difference between a 10-line fix that requires deep mathematical insight versus a 10-line fix that is just a tedious variable rename.
  • From-Scratch Generation: While the model became an expert at repairing code, the authors noted mixed results when asking the model to generate entire ML solutions from scratch, suggesting that "synthesis" requires different optimization targets than "repair."
  • Future Directions: The authors suggest expanding CCI to capture more semantic nuances and applying the BuggyMLE framework to even more diverse engineering domains beyond ML.

Summary updated

Abstract

Enabling AI systems to autonomously improve AI models requires agents capable of performing machine learning engineering (MLE). In practice, designing, training, and evaluating complex models demands modifying multi-file repositories to reproduce results, debug training runs, integrate datasets, and port frameworks. However, existing MLE benchmarks limit agents to single-file edits without full repository or shell access, and common repository-level software engineering benchmarks have rapidly saturated. To address this, we introduce the Code Complexity Index (CCI), a difficulty metric based on ten structural features spanning repository scale, dependency depth, and patch size. CCI strongly predicts average solve rates across eight public coding benchmarks ($R^2 = 0.951$), identifying patch size and complexity as top predictors. Guided by CCI, we develop BuggyMLE, a scalable framework where agents inject controlled mutations into working ML repositories, synthesizing over 6,600 bug-injected MLE repository variants and nearly 260K lines of injected code based on 222 source repositories. Using BuggyMLE, we make two contributions. First, we construct BuggyMLE-Bench-Lite, the first difficulty-graded, repository-level variant of MLE-Bench Lite; across 14 evaluated models, the top solve rate drops from 95% to 9% as difficulty is scaled from the easiest to the hardest tier. Second, we synthesize a training curriculum across three difficulty levels over 200 additional ML repositories, fine-tuning a Qwen3.5-9B model to rival frontier models on BuggyMLE-Bench-Lite. Our trained model transfers beyond ML, raising its SWE-bench Lite resolve rate from 24% to 33% and its LiveCodeBench v6 pass@1 from 69% to 71%.
BibTeX
@misc{wu2026buggymle,
  author = {John Wu and Rushi Qiang and Yuchen Zhuang and Guowang Li and Girish Narayanswamy and Max Xu and Sam Schmidgall and Mark Malhotra and Shwetak Patel and Jimeng Sun and Daniel McDuff and Bo Dai and Xin Liu},
  title = {{BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection}},
  year = {2026},
  eprint = {2611.25088},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25088},
}
Figure