DARE: Difficulty Analysis for Repair Enhancement

Jikwang Lee, Seungwook Jung, Jaechang Nam

TL;DR

DARE (Difficulty Analysis for Repair Enhancement) is a new framework that moves beyond simple "solve rates" to predict how hard a software repair task is for an LLM coding agent and why. By identifying specific bottlenecks—like difficult fault localization or complex code transformations—and providing targeted support, DARE boosted agent success rates from 54.7% to 79.6% on challenging real-world bugs.

What Problem Does This Paper Address?

Currently, LLM-based coding agents are evaluated primarily by their "solve rate" on benchmarks like SWE-bench. However, this metric is a blunt instrument. It doesn't distinguish between an agent that solves 50 easy tasks and one that solves 50 "impossible" ones. Furthermore, measuring task difficulty directly is prohibitively expensive, as it requires running dozens of different agent configurations on every single task to see how often they fail.

Without a way to quantify difficulty, researchers can't tell if a new model is actually smarter or if it's just better at the "low-hanging fruit." Moreover, a simple difficulty score doesn't tell a developer how to help the agent. Is the agent failing because it can't find the bug (localization) or because the fix requires a massive architectural change (transformation)?

Methods

The authors propose a three-stage framework: Extract, Predict & Analyze, and Apply Support.

  1. Difficulty Definition: They define difficulty as the failure rate of a task across a large population of diverse coding agents.
  2. Feature Extraction: They extract 151 static features from benchmark artifacts (issue descriptions, codebases, and tests). These are organized into seven Repair-Task Feature Groups:
  3. Specification (S): Clues in the issue description.
  4. Spec-Solution Gap (SG): How much the issue description differs from the final fix.
  5. Localization (L): How hard it is to find the files needing changes.
  6. Transformation (T): The complexity of the required code changes.
  7. Validation (V): The complexity of the tests needed to verify the fix.
  8. Context (C): Repository-wide characteristics.
  9. Interaction (INT): Combinations of features across groups.
  10. The DARE Model: They use a Generalized Additive Model (GAM). Unlike "black-box" neural networks, GAMs are inherently interpretable; the model's final difficulty prediction is the sum of contributions from individual features.
  11. Scale Alignment: Since different benchmarks use different agents, they use a "shared-agent scale alignment" (using GPT-4o and Claude 3.7 as anchors) to ensure difficulty scores are comparable across different datasets.
  12. Targeted Support: DARE identifies the "salient" feature group (the one contributing most to a high difficulty score) and attaches a specific assistance module (e.g., a Localization module or a Transformation module) to help the agent overcome that specific hurdle.
Fig. 1. Overview of the DARE framework. Static repair artifacts yield task features organized into seven Feature Groups. The DARE difficulty model predicts repair difficulty, while its feature contributions are aggregated by Feature Group to guide the selection of task-specific assistance. The displayed contribution values and selected module are illustrative.

Datasets

The study evaluates DARE on a massive collection of 2,594 tasks partitioned into four sets to ensure no data leakage:

  • SWE-bench Verified: 500 tasks reviewed by humans.
  • SWE-bench Lite \ Verified: 207 tasks.
  • SWE-bench Full \ (Verified ∪ Lite): 1,587 tasks.
  • SWE-bench-Live Lite: 300 more recent tasks.

Key Insights and Ideas

  • Difficulty is Universal: The authors found that task difficulty is remarkably consistent across different model families (OpenAI vs. Anthropic) and performance levels. If a task is hard for GPT-4o, it’s likely hard for Claude 3.5, suggesting a shared underlying structure to software repair difficulty.
  • The "Why" Matters: Two tasks can have the same high difficulty score for completely different reasons. As shown in Figure 3, at the highest difficulty level (L5), some tasks are hard because of "Transformation" (complex code changes), while others are hard because of "Specification-Solution Gap."
  • Interpretability is Actionable: By using an additive model (GAM), the authors could pinpoint exactly which category of features made a task hard. This allowed them to provide "just-in-time" help—like giving the agent extra tools for dependency analysis only when the model predicted the "Transformation" aspect was the bottleneck.
Distribution of salient feature groups across difficulty levels, showing that different factors dominate as tasks get harder.

Key Results

  • Accurate Prediction: DARE achieved a Spearman’s $\rho$ of .476 to .621 on held-out benchmark partitions, meaning it effectively ranked tasks from easiest to hardest without ever seeing the agent's actual performance on those tasks.
  • Meaningful Levels: The predicted difficulty levels (L1–L5) aligned closely with actual failure rates. For example, in the Verified set, the median failure rate jumped from 5.6% at L1 to 83.1% at L5.
  • Performance Boost: In a controlled experiment on 9 high-difficulty tasks, matching the support module to the task's salient feature group raised the success rate from 54.7% to 79.6% (a 24.9 percentage point improvement).
Boxplots showing the strong correlation between DARE's predicted difficulty levels and the observed failure rates across four different benchmark partitions.

Limitations and Future Work

  • Intervention Scale: While the difficulty prediction was tested on thousands of tasks, the actual "repair enhancement" (RQ3) was evaluated on a smaller set of 9 tasks (with 25 runs each). Broader validation is needed to confirm these gains hold across all task types.
  • Python-Centric: The current feature extractors and support modules are tailored for Python. Generalizing to other languages (like Java or C++) would require new static analysis tools.
  • Benchmark Dependency: Some features (like the "Specification-Solution Gap") rely on having a reference patch, which is available during benchmark creation but not during real-world deployment. The authors suggest DARE is currently best suited for benchmark developers and researchers evaluating agent progress.

Summary updated

Abstract

LLM-based coding agents are primarily evaluated by their solve rates. However, this metric does not reveal how difficult the solved tasks are or account for differences intask difficulty across benchmarks. Determining the difficulty of individual tasks by repeatedly running multiple agents is costly, while a predicted difficulty score alone provides limited guidance on what support an agent needs. To address these challenges, we propose DARE (Difficulty Analysis for Repair Enhancement), which integrates repair-task difficulty measurement, prediction, feature-group contribution analysis, and task-specific repair support. We define continuous task difficulty as the observed failure rate across multiple coding agents. The DARE difficulty model predicts this difficulty using 151 static features extracted from benchmark artifacts. The framework also analyzes the contributions of seven Repair-Task Feature Groups to identify the characteristics most salient to each task and uses this information to select task-specific agent support. We evaluate DARE on four benchmark partitions comprising 2,594 tasks. The difficulty model's aligned predictions achieve Spearman's $ρ$ = .476-.621 on held-out partitions, and the most salient Feature Groups vary across tasks and benchmarks. Furthermore, coding-agent experiments show improved repair success when support is matched to each task's salient Feature Group. These findings demonstrate that difficulty prediction and interpretation can lead to practical improvements in automated program repair (APR), while also providing a basis for difficulty-stratified agent evaluation, cross-benchmark comparisons, and identifying areas of repair capability that warrant further improvement.
BibTeX
@misc{lee2026dare,
  author = {Jikwang Lee and Seungwook Jung and Jaechang Nam},
  title = {{DARE: Difficulty Analysis for Repair Enhancement}},
  year = {2026},
  eprint = {2611.05797},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05797},
}
Figure