TL;DR
BuggyMLE is a new framework that synthetically scales the difficulty of machine learning engineering (MLE) tasks by injecting controlled "bugs" into real-world code repositories. By introducing the Code Complexity Index (CCI)—a structural metric that predicts task difficulty with remarkable accuracy ($R^2 = 0.951$)—the authors developed a curriculum that allowed a 9B-parameter model to rival frontier models like Claude 3.5 Opus and significantly improve its performance on general software engineering benchmarks.
What Problem Does This Paper Address?
As AI agents move from simple question-answering to autonomous research and engineering, we need better ways to evaluate them. Machine Learning Engineering (MLE) is a perfect "proving ground" because it requires navigating complex, multi-file codebases, running training pipelines, and optimizing measurable metrics.
However, current MLE benchmarks suffer from three major flaws:
- Oversimplification: They often focus on single-file edits, ignoring the "spaghetti" nature of real-world repositories.
- Saturation: Frontier models are already outperforming human reference solutions on existing suites.
- Curation Bottleneck: Manually creating high-quality, repository-level tasks with verified solutions is incredibly expensive and slow.
The authors ask: Can we automate the creation of difficult, realistic MLE tasks, and can we measure exactly how hard those tasks are?
Methods
The authors tackle this with a two-pronged approach: measuring difficulty and generating it.
1. The Code Complexity Index (CCI)
The researchers identified ten structural features of code—such as lines of code, dependency depth, and the number of "hunks" (blocks of modified lines) in a patch. They found that these structural properties alone, independent of the specific AI model being tested, are incredibly strong predictors of how likely a model is to solve a task.
2. The BuggyMLE Framework
To generate tasks, they built a four-stage pipeline:
- Injector: An LLM agent explores a "clean" ML repository and injects breaking changes (bugs) across data loading, model architecture, or training code. The "difficulty" is controlled by a budget of how many files and lines the agent is allowed to break.
- Oracle: An automated process that verifies which injected bugs actually degrade performance, creating a "ground truth" for what needs to be fixed.
- Solver: The agent under test attempts to diagnose and repair the repository using a standard terminal environment.
- Evaluator: A system that grades the repair based on metric recovery (e.g., does the model's accuracy return to baseline?) and a per-bug code review.
Datasets
The paper introduces BuggyMLEBench-Lite, a benchmark consisting of 88 oracle-verified tasks across four difficulty tiers (1, 5, 10, and 50 bugs per repository) derived from 22 diverse ML projects.
For training, they synthesized a massive dataset:
- 6,674 bug-injected variants based on 222 source repositories.
- ~260,000 lines of injected code.
- Training Curriculum: They used this data to fine-tune a Qwen3.5-9B model using Supervised Fine-Tuning (SFT) and Reinforcement Learning (specifically Group Relative Policy Optimization, or GRPO).
Key Insights and Ideas
- Structure Drives Difficulty: The most surprising finding is that structural features—especially patch size and complexity—explain 95% of the variance in solve rates across eight major public coding benchmarks. If a fix requires touching many files and hunks, it becomes exponentially harder for current models.
- The "CCI Slider": By using CCI, the authors can precisely tune the difficulty of a synthetic task. As they "slide" the CCI up by injecting more bugs, solve rates for every model tested (from small open-weight models to GPT-4o) drop predictably.
- MLE Training Generalizes to SWE: Training a model specifically to fix ML pipelines actually makes it a better general-purpose programmer. The researchers saw significant "transfer" to non-ML benchmarks like SWE-bench.
Key Results
- Benchmark Scaling: On BuggyMLEBench-Lite, the top-performing model's solve rate plummeted from 95% on the easiest tier (1 bug) to just 9% on the hardest tier (50 bugs), proving the benchmark is far from saturated.
- 9B Model vs. Flagships: Their fine-tuned Qwen3.5-9B model, trained on an "All Difficulties" curriculum, outperformed much larger models like Claude 3 Opus on the 10-bug tier.
- Out-of-Distribution Transfer: The model's performance on SWE-bench Lite (general software engineering) jumped from 24% to 33%, and its LiveCodeBench (competitive programming) score rose from 69% to 71%.
Limitations and Future Work
- Semantic vs. Structural Complexity: While CCI is a great predictor, it relies on structural features. It doesn't yet capture "semantic" difficulty—the difference between a 10-line fix that requires deep mathematical insight versus a 10-line fix that is just a tedious variable rename.
- From-Scratch Generation: While the model became an expert at repairing code, the authors noted mixed results when asking the model to generate entire ML solutions from scratch, suggesting that "synthesis" requires different optimization targets than "repair."
- Future Directions: The authors suggest expanding CCI to capture more semantic nuances and applying the BuggyMLE framework to even more diverse engineering domains beyond ML.
Summary updated