Abstract
Information Retrieval (IR) techniques underpin many critical Software Engineering (SE) tasks, yet the selection of an appropriate similarity measure is frequently treated as a secondary concern. This research investigates the performance of four prominent IR models—Vector Space Model (VSM), Latent Semantic Indexing (LSI), BM25, and Word Mover’s Distance (WMD)—across two distinct SE domains: project recommendation and bug localization. Our findings reveal that there is no universal "best" measure; instead, performance is highly dependent on the nature of the artifacts. LSI excels in natural language contexts, VSM is superior for structured code comparisons, and BM25 provides the most robust results for mixed-artifact tasks like bug localization.
Introduction
In the realm of software engineering, Information Retrieval (IR) serves as the engine for diverse activities, ranging from identifying similar repositories on platforms like GitHub to pinpointing the source files responsible for specific software defects. Central to these systems is the similarity measure—the mathematical framework that determines the proximity between a query and a document.
While models such as VSM and BM25 are staples in traditional natural language processing, SE artifacts present unique challenges. They typically consist of a heterogeneous mix of structured code (APIs, identifiers), semi-structured documentation (README files), and unstructured natural language (bug reports). This study addresses a fundamental gap: determining whether the choice of similarity measure significantly impacts SE task performance and identifying which models are best suited for specific data types.
Methodology
The researchers conducted a comparative analysis of four IR models:
- Vector Space Model (VSM): A traditional bag-of-words approach utilizing TF-IDF weighting and cosine similarity.
- Latent Semantic Indexing (LSI): A context-aware model using singular value decomposition to identify latent semantic relationships (e.g., synonyms).
- BM25 (Best Matching 25): A probabilistic model that incorporates document length normalization, widely used in modern search engines.
- Word Mover’s Distance (WMD): A semantic measure based on word embeddings (Word2Vec) that calculates the minimum "cost" to transform one document into another.
These models were evaluated on two representative SE tasks: project recommendation (finding similar GitHub projects) and bug localization (matching bug reports to source code).
Experimental Setup and Datasets
The study leveraged two primary datasets to evaluate the models across different artifact types.
The GitHub Project Dataset comprises 1,832 Java projects categorized into 112 functional groups. These categories were manually verified to ensure a high-quality ground truth for project recommendation.
The Bug Localization Dataset is a benchmark containing 1,100 bug reports from four major open-source projects: Birt, Eclipse-UI, JDT, and SWT. This dataset allows for testing the models' ability to bridge the gap between natural language queries and source code documents.
Discussion of Findings
The results highlight that "Code is not English." Context-aware models like LSI and WMD are effective for natural language because they recognize semantic similarities (e.g., "update" vs. "upgrade"). However, in source code, precision is paramount. When two projects share specific API calls, such as java.util.List, they are functionally similar; "fuzzy" semantic matching in these instances often introduces noise rather than clarity.
Furthermore, bug localization presents a "Mixed Artifact" challenge. It requires comparing concise, text-heavy bug reports with extensive, code-heavy source files. BM25 emerged as the superior choice here due to its robust document length normalization, which prevents longer source files from disproportionately influencing the results.
Interestingly, advanced embedding-based models like WMD struggled with GitHub descriptions. The researchers hypothesize that because these descriptions are often brief and keyword-dense, the semantic overhead of embeddings may actually degrade performance compared to simpler keyword matching.
Evaluation Results
- Text-to-Text Performance: For natural language project descriptions, LSI outperformed other models, achieving an 11.76% improvement in Mean Average Precision (MAP@10).
- Code-to-Code Performance: When comparing API names and package imports, VSM was the most effective, outperforming LSI by 20.59% to 24.29%.
- Mixed Artifact Performance: BM25 was the clear leader in bug localization. In the JDT project, BM25 achieved a staggering 366% increase in MAP@10 compared to the standard VSM.
- Practical Application: The study demonstrated that existing SE tools (such as CLAN and LR) could be significantly improved simply by swapping their internal similarity measures—replacing LSI with VSM in CLAN and VSM with BM25 in LR—without altering the tools' core architectures.
Conclusion and Future Work
While the study provides strong evidence for the impact of similarity measures, it is limited by the scope of the GitHub dataset and the potential for human bias in manual categorization. Future work should investigate hybrid models that apply different similarity measures to different components of a query—for instance, using LSI for the natural language portion of a bug report and VSM for embedded code snippets—to further refine retrieval accuracy.
Summary updated