Where's the Bug? Attention Probing for Scalable Fault Localization

Adam Stein, Arthur Wayne, Aaditya Naik, Mayur Naik, Eric Wong

TL;DR

Bug Attention Probe (BAP) is a lightweight fault localization method that pinpoints buggy lines of code without requiring explicit line-level labels or executable test cases. By training a simple attention probe on top of a small, frozen language model using only binary "buggy vs. clean" labels, BAP outperforms massive models like GPT-4o and 70B-parameter LLMs at a fraction of the computational cost.

What Problem Does This Paper Address?

Fault localization (FL)—identifying the exact line or lines of code causing a software bug—is a notoriously difficult and time-consuming task for developers. While automated tools exist, traditional approaches (like DeepFL) rely heavily on executable test cases, which are often unavailable or expensive to run continuously.

Recently, researchers have turned to Large Language Models (LLMs) to find bugs via zero-shot prompting. However, this strategy requires massive, resource-heavy models (such as GPT-4o or 70B-parameter open models) to achieve decent accuracy. Calling these giant models for every single function in a large codebase quickly becomes prohibitively slow and expensive. Alternative AI approaches require fine-grained, line-level bug annotations, which are rare, expensive to produce, and highly noisy.

The authors of this paper tackle a fundamental question: How can we achieve highly accurate fault localization without relying on executable tests, massive LLMs, or expensive line-level training labels?

Figure 1: Comparison of our approach Bug Attention Probe (BAP) with baselines DeepFL and LLM prompting on a Java program snippet. The program has two bugs: the age condition on line 3 is reversed and line 6 throws a null pointer exception. BAP correctly localizes both bugs. Here, our method is trained on Llama-3.2-1B, a “small” language model (SLM), with only weak supervision i.e. binary bug presence labels. Obtaining comparable accuracy via prompt- ing demands a significantly more resource-intensive LLM, such as Llama-3.2-90B, or even larger. Previous approaches to fault localization like DeepFL require executable test cases before they can attempt to provide useful information.

Methods

The authors introduce Bug Attention Probe (BAP), a technique rooted in weak supervision and attention probing. Instead of forcing an LLM to generate line numbers through prompting, BAP extracts localization directly from the internal mechanics of a small language model (SLM).

  1. Weak Supervision (Binary Classification): The authors take a small, frozen model (like Llama-3.2-1B or 11B) and add a single-layer Transformer decoder block on top of its intermediate hidden states. They train this lightweight probe on a simple task: predicting whether a given code snippet is overall "buggy" or "clean."
  2. Attention Probing: To successfully classify a program as buggy, the probe must internally figure out why it is buggy. BAP extracts the raw token-level attention scores from the probe relative to the final end-of-sequence [EOS] token.
  3. Line-Level Aggregation: Because raw tokens aren't intuitive for human developers, BAP sums up the token attention scores within each line of code. This yields a line-level "bugginess" score, allowing the system to rank lines from most to least suspicious.
Figure 2: Illustration of BAP as a method to elicit line-level fault localization from a frozen LLM through weak supervision. In step one, the probe is trained as a binary classifier to distinguish buggy from non-buggy code. Then in step two, we visualize the learned attention weights on the given sequence. Finally, in step three, we sum the attention weights within each line to produce a line-level “bugginess” score. BAP localizes the bug to the line with the highest score, the Top-1 result.

Datasets

BAP was evaluated across an extensive suite of eight diverse datasets spanning Python, Java, and C, encompassing over 50,000 code samples:

  • Defects4J (v1.2.0 & v3.0.1): The gold-standard benchmark containing real-world Java bugs and human-written fixes.
  • GitHub-Python & GitHub-Java: Code mined from GitHub containing real syntax errors.
  • DeepFix: A dataset of C programs written by students, containing common beginner syntax mistakes.
  • TSSB-3M & ManySStuBs4J: Python and Java datasets consisting of single-statement "simple, stupid" bugs mined from public repositories.
  • Juliet-Java & Juliet-C: Synthetic code blocks mapping to 89 different Common Weakness Enumerations (CWEs) to test security vulnerability localization.

Key Insights and Ideas

  • The "Aha!" of Weak Supervision: You do not need to explicitly tell a model where a bug is during training. Forcing a model to excel at bug detection naturally incentivizes its internal attention mechanisms to locate the root cause of the bug.
  • Small Models Can Outpunch Giants: By properly probing internal representations, a 1B or 11B parameter model can significantly outperform a 70B model or GPT-4o at zero-shot prompting.
  • Natural Multi-Line Handling: Prompting methods struggle to accurately output multiple buggy lines in sequence. Because BAP evaluates attention across all tokens in parallel, it naturally excels at identifying multi-line bugs or multiple independent bugs within a single function.

Key Results

  • State-of-the-Art Accuracy: Averaged across all eight datasets, BAP improved Top-1 localization accuracy by 34.6% over the strongest baseline and 93.4% over zero-shot prompting GPT-4o.
  • Massive Efficiency Gains: BAP built on top of Llama-3.2-11B achieved a 0.334 Top-1 accuracy on Defects4J, easily beating Llama-3.3-70B (0.269) while utilizing 10x fewer FLOPs and requiring a fraction of the GPU memory (24.2 GB vs 154.0 GB).
Figure 3: Model scale versus Top-1 on Defects4J. Each point for BAP is trained on the hidden representations from the Llama-3.2 model of the corresponding size.
  • Clear Interpretability: Qualitative visualizations demonstrate that BAP cleanly highlights the exact lines containing logical flaws or syntax errors, providing developers with clear, interpretable feedback.
Figure 4: Examples of bug localization with BAP on two evaluation set samples. We visualize the line-level weights from BAP above such that lines highlighted in a darker color have higher weights. BAP correctly identifies bug locations at Top-1.

Limitations and Future Work

  • The 50-Line Cliff: BAP's performance degrades sharply as the context length grows. For code fragments longer than 50 lines, all evaluated methods—including BAP and massive proprietary models—perform near random guessing.
Figure 5: Top-1 accuracy versus context length, measured by lines of code (LOC) on Defects4J. We compare BAP- Llama3.2-11B against models at least six times larger.
  • Future Directions: The authors highlight two critical areas for future work: improving length generalization so models can handle larger functions, and automatically scaling up weak-supervision datasets by running existing static bug detectors over massive code repositories.

Summary updated

Abstract

Ensuring code correctness remains a challenging problem even as large language models (LLMs) become increasingly capable at code-related tasks. While LLM-based program repair systems can propose bug fixes using only a user's bug report, their effectiveness is fundamentally limited by their ability to perform fault localization (FL), a challenging problem for both humans and LLMs. Existing FL approaches rely on executable test cases, require training on costly and often noisy line-level annotations, or demand resource-intensive LLMs. In this paper, we present Bug Attention Probe (BAP), a method which learns state-of-the-art fault localization without any direct localization labels, outperforming traditional FL baselines and prompting of large-scale LLMs. We evaluate our approach across a variety of code settings, including real-world Java bugs from the standard Defects4J dataset as well as seven other datasets which span a diverse set of bug types and languages. Averaged across all eight datasets, BAP improves by 34.6% top-1 accuracy compared to the strongest baseline and 93.4% over zero-shot prompting GPT-4o. BAP is also significantly more efficient than prompting, outperforming large open-weight models at a small fraction of the computational cost.
BibTeX
@misc{stein2025where,
  author = {Adam Stein and Arthur Wayne and Aaditya Naik and Mayur Naik and Eric Wong},
  title = {{Where's the Bug? Attention Probing for Scalable Fault Localization}},
  year = {2025},
  eprint = {2502.13966},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.13966},
}
Figure