TL;DR
This paper introduces Adaptive Multi-modal Bug Localization (AML), a novel technique that combines information retrieval (IR) from bug reports and program spectra from execution traces to pinpoint buggy code. AML significantly outperforms existing methods by adaptively learning the best way to weigh these different information sources for each bug, leading to more accurate bug localization.
What Problem Does This Paper Address?
Debugging is a time-consuming and resource-intensive task for software developers. A crucial part of debugging is bug localization: identifying the specific code elements that contain a bug. Existing approaches typically fall into two categories: Information Retrieval (IR)-based methods that analyze the text of bug reports, and spectrum-based methods that analyze program execution traces (which parts of the code ran during successful and failing tests). While both have their strengths, they only leverage one type of information. This is suboptimal because hints about a bug's location can be scattered across both textual descriptions and execution patterns. Developers often need to manually sift through code, and a single bug might leave traces in both the bug report and the program's behavior. This paper addresses the limitation of single-source approaches by proposing a multi-modal technique that intelligently combines both bug reports and program spectra.
Methods
The core of the proposed approach is Adaptive Multi-modal Bug Localization (AML). AML is designed to be adaptive, meaning it learns how to best combine different information sources for each specific bug, rather than using a one-size-fits-all strategy. It comprises three main components:
- AMLText: This component uses traditional IR techniques (like Vector Space Model with TF-IDF) to analyze the textual content of bug reports and find similarities with program elements (methods in this case).
- AMLSpectra: This component uses spectrum-based bug localization techniques (like Tarantula) to analyze program execution traces from failing and successful test cases, identifying program elements that are more frequently executed in failing scenarios.
- AMLSuspWord: This is a novel component that introduces the concept of "suspicious words." It identifies words within bug reports and code that are highly associated with a bug by considering their co-occurrence with program elements in both failing and successful execution traces. These suspicious words are then used to compute a method's suspiciousness score.
The Integrator component then combines the scores from AMLText, AMLSpectra, and AMLSuspWord using a weighted sum. Crucially, the weights (α, β, γ) for these components are learned adaptively for each new bug. This learning process uses a probabilistic approach and a balanced sampling strategy to handle the skewed distribution of faulty vs. non-faulty methods, fine-tuning the weights based on a set of similar historical bugs.
Datasets
The authors evaluated their approach on a dataset of 157 real bugs collected from four popular software projects: AspectJ, Ant, Lucene, and Rhino. These projects are described as medium to large scale and implemented in Java. For AspectJ, bugs were sourced from the iBugs dataset. The remaining bugs from Ant, Lucene, and Rhino were collected by the authors. For each bug, they had access to the pre-fix version of the code, a set of successful test cases, and at least one failing test case.
Key Insights and Ideas
- Adaptivity is Key: The most significant insight is that a "one-size-fits-all" approach to bug localization is suboptimal. Different bugs have different characteristics, and AML's ability to adapt its weighting strategy for each bug is crucial for its superior performance.
- Suspicious Words Matter: The introduction of the "suspicious words" concept (AMLSuspWord) is a novel contribution. This idea acknowledges that specific terms within code and bug reports can be strong indicators of a bug's location, and leveraging this provides a new dimension to bug localization.
- Combining Sources is Better: The paper strongly supports the hypothesis that combining textual information from bug reports with execution trace data from program spectra leads to more accurate bug localization than using either source alone.
- Developer-Centric Evaluation: The evaluation metrics (Top N and Mean Average Precision) are chosen to reflect how a developer would actually use these tools – inspecting a ranked list of potential bug locations.
Key Results
AML significantly outperformed all baseline methods across various metrics:
- Top N Accuracy: AML successfully localized 31, 71, and 92 bugs when developers inspected the top 1, 5, and 10 methods, respectively. This represents an improvement of 47.62%, 31.48%, and 27.78% over the best baseline (PROMESIR) for these respective Top N values.
- Mean Average Precision (MAP): AML achieved an overall MAP score of 0.237, outperforming the best baseline by 28.80%.
- Component Contributions: The AMLSuspWord component was found to be particularly important, and omitting any single component (AMLText, AMLSpectra, or AMLSuspWord) reduced the overall effectiveness of AML.
- Running Time: The average running time for AML to produce a ranked list of methods was 46.01 seconds, which the authors deem reasonable for a developer's debugging workflow.
Limitations and Future Work
- Threats to Validity: The authors acknowledge potential threats to internal validity (implementation/dataset errors) and external validity (generalizability to other programming languages or larger/different types of software). They plan to address external validity by testing on more diverse systems.
- Dataset Size: While 157 bugs is a substantial dataset, future work could involve analyzing even more bugs to further validate the approach.
- Temporal Ordering: The cross-validation approach used for evaluation does not account for the temporal ordering of bugs, which could be a factor in real-world scenarios.
- Future Work: The authors plan to improve AML's effectiveness further and investigate its performance on more diverse software systems and programming languages. They also suggest exploring alternative machine learning algorithms for the integrator component.
Summary updated