Large Language Models (LLMs) have achieved remarkable success in short-form code generation, such as completing isolated functions or translating natural language into simple SQL queries. However, a profound gap exists between writing a clean snippet of code from scratch and debugging a broken, large-scale software system.
When an LLM generates code, it maps a local prompt to a local syntax tree. But debugging a real-world codebase requires something entirely different: constructing a global dependency graph, tracing execution state over time, and isolating a specific root cause from a noisy symptom. Recent research reveals that as software complexity scales from single functions to entire repositories, the performance of state-of-the-art models plummets, exposing fundamental limitations in how AI reasons about complex software systems.
The Needle in the Haystack: Fault Localization
The first and often most difficult step in debugging is figuring out where to look. In a massive repository containing thousands of files, a bug might manifest as a failure in an integration test, but the root cause could be buried deep within an unrelated utility module.
[Repository-Scale Codebase]
│
├── src/auth/ ─── (Root Cause: Silent logic error)
│
└── src/utils/ ─── [Test Failure Symptom]
This needle-in-a-haystack problem severely challenges current models. In their work introducing the SWE-bench benchmark, Carlos E Jimenez et al. demonstrated that language models struggle significantly with bug localization in large-scale repositories. Identifying the correct file to edit is often just as difficult as generating the fix itself. Surprisingly, the authors found that even when the relevant files are pre-selected and handed to the model, the success rate for resolving real-world GitHub issues remains extremely low. This indicates that the challenge is not merely one of context window size, but of deep contextual understanding.
The Geometry of Real-World Bugs
Why does repository-level debugging break the standard capabilities of LLMs? Real-world software defects possess a structural complexity that cannot be easily flattened into a sequence of text tokens. Two specific topologies highlight this issue: multi-hop errors and multi-bug interactions.
Multi-Hop Errors
In simple programming assignments, an error usually triggers an exception exactly where the logic is broken (e.g., a division by zero). In complex applications, however, errors are often "multi-hop." As Zhiyu Yang et al. explain in their study of data science code debugging, a multi-hop error occurs when the root cause is chronologically or spatially distant from the observed symptom.
For example, an off-by-one error in a data preprocessing step might silently alter the dimensions of an array. The program continues to run smoothly across several modules until, five steps later, a matrix multiplication fails. To debug this, an agent cannot simply look at the crash site; it must simulate the execution flow backward through time. The authors show that LLMs struggle immensely with this backward tracing, revealing a core limitation in their ability to simulate complex program behavior.
Multi-Bug Interactions
Real codebases frequently suffer from multiple concurrent logical flaws that mask or compound one another. When tasked with debugging code containing multiple interacting errors, LLMs demonstrate a dramatic reduction in accuracy. Zhiyu Yang et al. observe that models fail to effectively disentangle these concurrent flaws. An LLM might successfully identify and fix one bug, but because the remaining bugs alter the downstream symptoms, the model becomes confused, enters a loop, or introduces new regressions.
The Web of Interdependencies
As software grows, its complexity shifts from the logic within functions to the relationships between functions, files, and external libraries.
[File A] ──(imports)──> [File B] ──(calls API)──> [External Library]
│ │
└──────────────────(leaks state to)───────────────────┘
Evaluating models on this axis requires benchmarks that mirror industrial realities. Jingjing Liu et al. introduced RepoDebug to evaluate multi-file and multi-language debugging, finding that state-of-the-art models consistently fail when required to manage complex logic spread across different parts of a codebase. Similarly, Tao Sun et al. developed RepoFixEval to track the entire pipeline from issue discovery to multi-file fixing. Their experiments confirm that advanced models like GPT-4o can only resolve a small fraction of real-world repository bugs, highlighting a massive gap between AI capabilities and practical software engineering demands.
This dependency problem extends beyond a repository's internal files to its external ecosystem. Real-world software relies heavily on third-party packages. Jinyang Huang et al. investigated how LLMs handle multi-library scenarios, discovering that models struggle to grasp the intricate relationships and interactions between varied, interconnected software components. When a bug arises from a subtle mismatch between how two different libraries handle state, static text patterns fail to guide the model to a solution.
Domain-Specific Scale: Kernels and Enterprise Data
The breakdown of LLM debugging is even more pronounced in specialized environments:
- Enterprise Data Engineering: Jing Ye et al. looked beyond standard "Text-to-SQL" tasks to evaluate how LLMs handle massive enterprise ETL (Extract, Transform, Load) SQL scripts. They found that while models excel at generating short queries, they fail to resolve more than a third of real-world debugging tasks when scripts span hundreds of lines with deeply nested dependencies.
- Operating System Kernels: At the lowest level of the stack, Zhenhao Zhou et al. benchmarked LLM agents on the Linux kernel. They noted that the kernel's massive scale, limited runtime observability, and intricate hardware interactions create an environment where general-purpose LLM debugging strategies completely break down.
Bridging the Gap: From Static Text to Dynamic Feedback
If static text generation is insufficient for repository-level debugging, how can the field move forward? The answer lies in combining the pattern-matching strengths of language models with the dynamic feedback loops of traditional software engineering tools.
Historically, Automated Program Repair (APR) tools relied on rigid, hardcoded semantic rules. As Zhiyu Fan et al. point out, traditional APR tools face severe limitations when dealing with complex bugs, multi-line fixes, and a lack of awareness regarding program dependencies—the exact types of errors common in LLM outputs. However, LLMs can be used to supercharge these traditional workflows, and vice versa.
Rather than treating a failed patch as a waste of compute, Yiling Lou et al. propose a unified debugging approach called ProFL. ProFL leverages program repair techniques to refine fault localization. By executing automatically generated patches and analyzing the resulting test failures, the system learns more about the nature of the bug, pinpointing the code location more effectively than static analysis alone.
┌──────────────┐ Executes ┌─────────────────┐
│ LLM Agent │ ───────────────> │ Compiler/Tester │
└──────────────┘ └─────────────────┘
▲ │
│ Feedback (Error Logs) │
└───────────────────────────────────┘
Grounding language models in dynamic execution loops yields significant performance gains. For instance, Azmain Kabir et al. developed ZS4C, a framework that uses an iterative compiler-LLM feedback loop to synthesize compilable code from incomplete snippets found on Q&A sites. By letting the compiler correct the LLM's assumptions about imports and types, the system achieves a much higher success rate.
Ultimately, the most powerful debugging paradigms bypass static code analysis entirely to look at runtime behavior. Xiang (Jenny) Ren et al. introduced "Relational Debugging" via a tool called Perspect. Instead of guessing where a bug is based on source text, Perspect requires only two executions: a "good" run and a "bad" run. By automatically analyzing the relationships between runtime events across these executions, it pinpoints root causes that traditional methods miss—especially performance anomalies that do not alter program predicates or throw explicit errors.
Conclusion
The transition from code generation to repository-level debugging marks a shift from syntax to causality. To close the gap between AI assistants and human software engineers, future systems cannot rely solely on larger context windows or scaling laws. They must integrate language models into tight, iterative loops with compilers, test suites, and relational runtime analyzers, transforming LLMs from static text writers into dynamic, hypothesis-driven debuggers.
Key Foundational Work
Automated Repair of Programs from Large Language Models (2023) — contributed 3 concept elements to this direction.
Can automated program repair refine fault localization? a unified debugging approach (2020) — contributed 1 concept elements to this direction.
Relational Debugging - Pinpointing Root Causes of Performance Problems (2023) — contributed 1 concept elements to this direction.