TL;DR
DARE (Difficulty Analysis for Repair Enhancement) is a new framework that moves beyond simple "solve rates" to predict how hard a software repair task is for an LLM coding agent and why. By identifying specific bottlenecks—like difficult fault localization or complex code transformations—and providing targeted support, DARE boosted agent success rates from 54.7% to 79.6% on challenging real-world bugs.
What Problem Does This Paper Address?
Currently, LLM-based coding agents are evaluated primarily by their "solve rate" on benchmarks like SWE-bench. However, this metric is a blunt instrument. It doesn't distinguish between an agent that solves 50 easy tasks and one that solves 50 "impossible" ones. Furthermore, measuring task difficulty directly is prohibitively expensive, as it requires running dozens of different agent configurations on every single task to see how often they fail.
Without a way to quantify difficulty, researchers can't tell if a new model is actually smarter or if it's just better at the "low-hanging fruit." Moreover, a simple difficulty score doesn't tell a developer how to help the agent. Is the agent failing because it can't find the bug (localization) or because the fix requires a massive architectural change (transformation)?
Methods
The authors propose a three-stage framework: Extract, Predict & Analyze, and Apply Support.
- Difficulty Definition: They define difficulty as the failure rate of a task across a large population of diverse coding agents.
- Feature Extraction: They extract 151 static features from benchmark artifacts (issue descriptions, codebases, and tests). These are organized into seven Repair-Task Feature Groups:
- Specification (S): Clues in the issue description.
- Spec-Solution Gap (SG): How much the issue description differs from the final fix.
- Localization (L): How hard it is to find the files needing changes.
- Transformation (T): The complexity of the required code changes.
- Validation (V): The complexity of the tests needed to verify the fix.
- Context (C): Repository-wide characteristics.
- Interaction (INT): Combinations of features across groups.
- The DARE Model: They use a Generalized Additive Model (GAM). Unlike "black-box" neural networks, GAMs are inherently interpretable; the model's final difficulty prediction is the sum of contributions from individual features.
- Scale Alignment: Since different benchmarks use different agents, they use a "shared-agent scale alignment" (using GPT-4o and Claude 3.7 as anchors) to ensure difficulty scores are comparable across different datasets.
- Targeted Support: DARE identifies the "salient" feature group (the one contributing most to a high difficulty score) and attaches a specific assistance module (e.g., a Localization module or a Transformation module) to help the agent overcome that specific hurdle.
Datasets
The study evaluates DARE on a massive collection of 2,594 tasks partitioned into four sets to ensure no data leakage:
- SWE-bench Verified: 500 tasks reviewed by humans.
- SWE-bench Lite \ Verified: 207 tasks.
- SWE-bench Full \ (Verified ∪ Lite): 1,587 tasks.
- SWE-bench-Live Lite: 300 more recent tasks.
Key Insights and Ideas
- Difficulty is Universal: The authors found that task difficulty is remarkably consistent across different model families (OpenAI vs. Anthropic) and performance levels. If a task is hard for GPT-4o, it’s likely hard for Claude 3.5, suggesting a shared underlying structure to software repair difficulty.
- The "Why" Matters: Two tasks can have the same high difficulty score for completely different reasons. As shown in Figure 3, at the highest difficulty level (L5), some tasks are hard because of "Transformation" (complex code changes), while others are hard because of "Specification-Solution Gap."
- Interpretability is Actionable: By using an additive model (GAM), the authors could pinpoint exactly which category of features made a task hard. This allowed them to provide "just-in-time" help—like giving the agent extra tools for dependency analysis only when the model predicted the "Transformation" aspect was the bottleneck.
Key Results
- Accurate Prediction: DARE achieved a Spearman’s $\rho$ of .476 to .621 on held-out benchmark partitions, meaning it effectively ranked tasks from easiest to hardest without ever seeing the agent's actual performance on those tasks.
- Meaningful Levels: The predicted difficulty levels (L1–L5) aligned closely with actual failure rates. For example, in the Verified set, the median failure rate jumped from 5.6% at L1 to 83.1% at L5.
- Performance Boost: In a controlled experiment on 9 high-difficulty tasks, matching the support module to the task's salient feature group raised the success rate from 54.7% to 79.6% (a 24.9 percentage point improvement).
Limitations and Future Work
- Intervention Scale: While the difficulty prediction was tested on thousands of tasks, the actual "repair enhancement" (RQ3) was evaluated on a smaller set of 9 tasks (with 25 runs each). Broader validation is needed to confirm these gains hold across all task types.
- Python-Centric: The current feature extractors and support modules are tailored for Python. Generalizing to other languages (like Java or C++) would require new static analysis tools.
- Benchmark Dependency: Some features (like the "Specification-Solution Gap") rely on having a reference patch, which is available during benchmark creation but not during real-world deployment. The authors suggest DARE is currently best suited for benchmark developers and researchers evaluating agent progress.
Summary updated