Towards Automating Precision Studies of Clone Detectors

Vaibhav Saini, Farima Farmahinifarahani, Yadong Lu, Di Yang, Pedro Martins, Hitesh Sajnani, Pierre Baldi, Cristina V. Lopes

Abstract: Current research in clone detection suffers from poor ecosystems for evaluating precision of clone detection tools. Corpora of labeled clones are scarce and incomplete, making evaluation labor intensive and idiosyncratic, and limiting inter tool comparison. Precision-assessment tools are simply lacking. We present a semi-automated approach to facilitate precision studies of clone detection tools. The approach merges automatic mechanisms of clone classification with manual validation of clone pairs. We demonstrate that the proposed automatic approach has a very high precision and it significantly reduces the number of clone pairs that need human validation during precision experiments. Moreover, we aggregate the individual effort of multiple teams into a single evolving dataset of labeled clone pairs, creating an important asset for software clone research.

Introduction

In software engineering, "code clones"—fragments of source code that are highly similar—are a double-edged sword. While they can facilitate rapid development, they often lead to increased maintenance costs, bug propagation, and complex licensing issues. Consequently, detecting these clones is vital. However, evaluating the tools designed to find them has historically been a lopsided endeavor. While measuring recall (the ability to find existing clones) is largely automated through benchmarks like BigCloneBench, measuring precision (the accuracy of reported clones) remains a grueling, manual bottleneck.

Assessing precision currently requires human "judges" to manually inspect thousands of code pairs. This process is labor-intensive, subjective, and the resulting datasets are rarely shared, forcing every new study to start from scratch. As codebases grow, the number of potential pairs to check increases quadratically, creating a combinatorial problem that quickly becomes infeasible for human researchers. This paper introduces InspectorClone, a semi-automated framework designed to reduce this manual workload by up to 84% while maintaining near-perfect precision.

The InspectorClone Methodology

InspectorClone utilizes a multi-stage "resolution pipeline" to classify clones automatically. A core philosophy of the system is prioritizing precision over recall: the authors intentionally set strict thresholds so that the machine only makes a "clone" call when it is highly certain, leaving ambiguous cases for human review.

The InspectorClone automatic resolution pipeline.

The pipeline consists of three primary automated stages:

  1. Type I Resolution (Exact Clones): The system strips comments and whitespace, then uses SHA-256 string hashing. If the hashes match, the pair is an exact clone.
  2. Type II Resolution (Renamed Clones): This stage targets clones where identifiers or literals have been changed. It utilizes "Action Tokens" (semantic signatures of method calls and field accesses) and 24 software metrics, such as cyclomatic complexity and nesting depth. If both the action tokens and metrics match 100%, it is classified as a clone.
  3. Type III Resolution (Near-miss Clones): This is the most difficult category, involving statement additions or deletions. To handle this, the authors developed a specialized deep learning architecture.

Type III Resolution: Siamese Deep Neural Networks

For near-miss clones, the authors trained a Siamese Deep Neural Network (DNN). This architecture is specifically optimized for comparison tasks. Unlike standard neural networks that process a single input, a Siamese network processes two inputs through identical subnetworks that share the same weights.

Architecture of the Siamese DNN used for Type III clone resolution.

The network takes the 24 software metrics from two code fragments, passes them through the twin subnetworks to map them into a feature space, and uses a "comparator" unit to output a similarity probability. By using a strict confidence threshold of 0.9, the model ensures that automated resolutions are highly reliable. This approach allows the model to learn a mapping where similar code fragments are physically close to each other in a high-dimensional vector space, making it significantly more effective than standard logistic regression or shallow networks.

Experimental Results and Impact

The system was trained on a massive dataset of 106,116 pairs (53,058 true clones and 53,058 "hard" non-clones) sourced from BigCloneBench and the IJaDataset-2.0. To ensure the model could distinguish between subtle differences, the authors used the Oreo tool to find non-clones that share high semantic similarity but different structures.

Key findings include: * Effort Reduction: InspectorClone reduced the number of pairs requiring manual inspection by an average of 39% across seven different clone detection tools. For some tools, like iClones, the reduction was as high as 84%. * High Automated Precision: The automated classification achieved a precision of 99.96%, with only one false positive identified out of 2,545 automatically resolved pairs. * DNN Performance: The Siamese DNN outperformed other models, achieving a precision of 0.996 and a recall of 0.700 on the test set.

Conclusion and Future Work

InspectorClone represents a significant step toward a more scalable ecosystem for clone detection research. By automating the "easy" and "moderately difficult" cases, it allows human experts to focus their limited time on the most ambiguous pairs. Furthermore, by aggregating individual validation efforts into a single, evolving dataset, it creates a permanent community asset that prevents researchers from repeating manual labor. While currently specialized for Java method-level clones, future work aims to expand the framework to other languages like C++ and Python, as well as different code granularities.

Summary updated

Abstract

Current research in clone detection suffers from poor ecosystems for evaluating precision of clone detection tools. Corpora of labeled clones are scarce and incomplete, making evaluation labor intensive and idiosyncratic, and limiting inter tool comparison. Precision-assessment tools are simply lacking. We present a semi-automated approach to facilitate precision studies of clone detection tools. The approach merges automatic mechanisms of clone classification with manual validation of clone pairs. We demonstrate that the proposed automatic approach has a very high precision and it significantly reduces the number of clone pairs that need human validation during precision experiments. Moreover, we aggregate the individual effort of multiple teams into a single evolving dataset of labeled clone pairs, creating an important asset for software clone research.
BibTeX
@inproceedings{saini2019towards,
  author = {Vaibhav Saini and Farima Farmahinifarahani and Yadong Lu and Di Yang and Pedro Martins and Hitesh Sajnani and Pierre Baldi and Cristina V. Lopes},
  title = {{Towards Automating Precision Studies of Clone Detectors}},
  booktitle = {ICSE},
  year = {2019},
  url = {https://openreview.net/forum?id=fbmGFMQu4j1},
}
Figure