TL;DR
Bug Attention Probe (BAP) is a lightweight fault localization method that pinpoints buggy lines of code without requiring explicit line-level labels or executable test cases. By training a simple attention probe on top of a small, frozen language model using only binary "buggy vs. clean" labels, BAP outperforms massive models like GPT-4o and 70B-parameter LLMs at a fraction of the computational cost.
What Problem Does This Paper Address?
Fault localization (FL)—identifying the exact line or lines of code causing a software bug—is a notoriously difficult and time-consuming task for developers. While automated tools exist, traditional approaches (like DeepFL) rely heavily on executable test cases, which are often unavailable or expensive to run continuously.
Recently, researchers have turned to Large Language Models (LLMs) to find bugs via zero-shot prompting. However, this strategy requires massive, resource-heavy models (such as GPT-4o or 70B-parameter open models) to achieve decent accuracy. Calling these giant models for every single function in a large codebase quickly becomes prohibitively slow and expensive. Alternative AI approaches require fine-grained, line-level bug annotations, which are rare, expensive to produce, and highly noisy.
The authors of this paper tackle a fundamental question: How can we achieve highly accurate fault localization without relying on executable tests, massive LLMs, or expensive line-level training labels?
Methods
The authors introduce Bug Attention Probe (BAP), a technique rooted in weak supervision and attention probing. Instead of forcing an LLM to generate line numbers through prompting, BAP extracts localization directly from the internal mechanics of a small language model (SLM).
- Weak Supervision (Binary Classification): The authors take a small, frozen model (like Llama-3.2-1B or 11B) and add a single-layer Transformer decoder block on top of its intermediate hidden states. They train this lightweight probe on a simple task: predicting whether a given code snippet is overall "buggy" or "clean."
- Attention Probing: To successfully classify a program as buggy, the probe must internally figure out why it is buggy. BAP extracts the raw token-level attention scores from the probe relative to the final end-of-sequence
[EOS]token. - Line-Level Aggregation: Because raw tokens aren't intuitive for human developers, BAP sums up the token attention scores within each line of code. This yields a line-level "bugginess" score, allowing the system to rank lines from most to least suspicious.
Datasets
BAP was evaluated across an extensive suite of eight diverse datasets spanning Python, Java, and C, encompassing over 50,000 code samples:
- Defects4J (v1.2.0 & v3.0.1): The gold-standard benchmark containing real-world Java bugs and human-written fixes.
- GitHub-Python & GitHub-Java: Code mined from GitHub containing real syntax errors.
- DeepFix: A dataset of C programs written by students, containing common beginner syntax mistakes.
- TSSB-3M & ManySStuBs4J: Python and Java datasets consisting of single-statement "simple, stupid" bugs mined from public repositories.
- Juliet-Java & Juliet-C: Synthetic code blocks mapping to 89 different Common Weakness Enumerations (CWEs) to test security vulnerability localization.
Key Insights and Ideas
- The "Aha!" of Weak Supervision: You do not need to explicitly tell a model where a bug is during training. Forcing a model to excel at bug detection naturally incentivizes its internal attention mechanisms to locate the root cause of the bug.
- Small Models Can Outpunch Giants: By properly probing internal representations, a 1B or 11B parameter model can significantly outperform a 70B model or GPT-4o at zero-shot prompting.
- Natural Multi-Line Handling: Prompting methods struggle to accurately output multiple buggy lines in sequence. Because BAP evaluates attention across all tokens in parallel, it naturally excels at identifying multi-line bugs or multiple independent bugs within a single function.
Key Results
- State-of-the-Art Accuracy: Averaged across all eight datasets, BAP improved Top-1 localization accuracy by 34.6% over the strongest baseline and 93.4% over zero-shot prompting GPT-4o.
- Massive Efficiency Gains: BAP built on top of Llama-3.2-11B achieved a 0.334 Top-1 accuracy on Defects4J, easily beating Llama-3.3-70B (0.269) while utilizing 10x fewer FLOPs and requiring a fraction of the GPU memory (24.2 GB vs 154.0 GB).
- Clear Interpretability: Qualitative visualizations demonstrate that BAP cleanly highlights the exact lines containing logical flaws or syntax errors, providing developers with clear, interpretable feedback.
Limitations and Future Work
- The 50-Line Cliff: BAP's performance degrades sharply as the context length grows. For code fragments longer than 50 lines, all evaluated methods—including BAP and massive proprietary models—perform near random guessing.
- Future Directions: The authors highlight two critical areas for future work: improving length generalization so models can handle larger functions, and automatically scaling up weak-supervision datasets by running existing static bug detectors over massive code repositories.
Summary updated