A Case Study on the Impact of Similarity Measure on Information Retrieval based Software Engineering Tasks

Md. Masudur Rahman, Saikat Chakraborty, Gail E. Kaiser, Baishakhi RayarXiv 2018 · OpenReview

Abstract

Information Retrieval (IR) techniques underpin many critical Software Engineering (SE) tasks, yet the selection of an appropriate similarity measure is frequently treated as a secondary concern. This research investigates the performance of four prominent IR models—Vector Space Model (VSM), Latent Semantic Indexing (LSI), BM25, and Word Mover’s Distance (WMD)—across two distinct SE domains: project recommendation and bug localization. Our findings reveal that there is no universal "best" measure; instead, performance is highly dependent on the nature of the artifacts. LSI excels in natural language contexts, VSM is superior for structured code comparisons, and BM25 provides the most robust results for mixed-artifact tasks like bug localization.

Introduction

In the realm of software engineering, Information Retrieval (IR) serves as the engine for diverse activities, ranging from identifying similar repositories on platforms like GitHub to pinpointing the source files responsible for specific software defects. Central to these systems is the similarity measure—the mathematical framework that determines the proximity between a query and a document.

While models such as VSM and BM25 are staples in traditional natural language processing, SE artifacts present unique challenges. They typically consist of a heterogeneous mix of structured code (APIs, identifiers), semi-structured documentation (README files), and unstructured natural language (bug reports). This study addresses a fundamental gap: determining whether the choice of similarity measure significantly impacts SE task performance and identifying which models are best suited for specific data types.

Methodology

The researchers conducted a comparative analysis of four IR models:

  1. Vector Space Model (VSM): A traditional bag-of-words approach utilizing TF-IDF weighting and cosine similarity.
  2. Latent Semantic Indexing (LSI): A context-aware model using singular value decomposition to identify latent semantic relationships (e.g., synonyms).
  3. BM25 (Best Matching 25): A probabilistic model that incorporates document length normalization, widely used in modern search engines.
  4. Word Mover’s Distance (WMD): A semantic measure based on word embeddings (Word2Vec) that calculates the minimum "cost" to transform one document into another.

These models were evaluated on two representative SE tasks: project recommendation (finding similar GitHub projects) and bug localization (matching bug reports to source code).

Experimental Setup and Datasets

The study leveraged two primary datasets to evaluate the models across different artifact types.

Statistics of the GitHub Project Dataset

The GitHub Project Dataset comprises 1,832 Java projects categorized into 112 functional groups. These categories were manually verified to ensure a high-quality ground truth for project recommendation.

Statistics of the Bug Localization Dataset

The Bug Localization Dataset is a benchmark containing 1,100 bug reports from four major open-source projects: Birt, Eclipse-UI, JDT, and SWT. This dataset allows for testing the models' ability to bridge the gap between natural language queries and source code documents.

Discussion of Findings

The results highlight that "Code is not English." Context-aware models like LSI and WMD are effective for natural language because they recognize semantic similarities (e.g., "update" vs. "upgrade"). However, in source code, precision is paramount. When two projects share specific API calls, such as java.util.List, they are functionally similar; "fuzzy" semantic matching in these instances often introduces noise rather than clarity.

Furthermore, bug localization presents a "Mixed Artifact" challenge. It requires comparing concise, text-heavy bug reports with extensive, code-heavy source files. BM25 emerged as the superior choice here due to its robust document length normalization, which prevents longer source files from disproportionately influencing the results.

Interestingly, advanced embedding-based models like WMD struggled with GitHub descriptions. The researchers hypothesize that because these descriptions are often brief and keyword-dense, the semantic overhead of embeddings may actually degrade performance compared to simpler keyword matching.

Evaluation Results

  • Text-to-Text Performance: For natural language project descriptions, LSI outperformed other models, achieving an 11.76% improvement in Mean Average Precision (MAP@10).
  • Code-to-Code Performance: When comparing API names and package imports, VSM was the most effective, outperforming LSI by 20.59% to 24.29%.
  • Mixed Artifact Performance: BM25 was the clear leader in bug localization. In the JDT project, BM25 achieved a staggering 366% increase in MAP@10 compared to the standard VSM.
  • Practical Application: The study demonstrated that existing SE tools (such as CLAN and LR) could be significantly improved simply by swapping their internal similarity measures—replacing LSI with VSM in CLAN and VSM with BM25 in LR—without altering the tools' core architectures.

Conclusion and Future Work

While the study provides strong evidence for the impact of similarity measures, it is limited by the scope of the GitHub dataset and the potential for human bias in manual categorization. Future work should investigate hybrid models that apply different similarity measures to different components of a query—for instance, using LSI for the natural language portion of a bug report and VSM for embedded code snippets—to further refine retrieval accuracy.

Summary updated

Abstract

Information Retrieval (IR) plays a pivotal role in diverse Software Engineering (SE) tasks, e.g., bug localization and triaging, code retrieval, requirements analysis, etc. The choice of similarity measure is the core component of an IR technique. The performance of any IR method critically depends on selecting an appropriate similarity measure for the given application domain. Since different SE tasks operate on different document types like bug reports, software descriptions, source code, etc. that often contain non-standard domain-specific vocabulary, it is essential to understand which similarity measures work best for different SE documents. This paper presents two case studies on the effect of different similarity measure on various SE documents w.r.t. two tasks: (i) project recommendation: finding similar GitHub projects and (ii) bug localization: retrieving buggy source file(s) correspond to a bug report. These tasks contain a diverse combination of textual (i.e. description, readme) and code (i.e. source code, API, import package) artifacts. We observe that the performance of IR models varies when applied to different artifact types. We find that, in general, the context-aware models achieve better performance on textual artifacts. In contrast, simple keyword-based bag-of-words models perform better on code artifacts. On the other hand, the probabilistic ranking model BM25 performs better on a mixture of text and code artifacts. We further investigate how such an informed choice of similarity measure impacts the performance of SE tools. In particular, we analyze two previously proposed tools for project recommendation and bug localization tasks, which leverage diverse software artifacts, and observe that an informed choice of similarity measure indeed leads to improved performance of the existing SE tools.
BibTeX
@misc{rahman2018case,
  author = {Md. Masudur Rahman and Saikat Chakraborty and Gail E. Kaiser and Baishakhi Ray},
  title = {{A Case Study on the Impact of Similarity Measure on Information Retrieval based Software Engineering Tasks}},
  year = {2018},
  eprint = {1808.02911},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.02911},
}
Figure