The Adverse Effects of Code Duplication in Machine Learning Models of Code

Miltiadis Allamanis

Abstract

Machine learning models for code are frequently evaluated on datasets containing significant amounts of near-duplicate files. This leads to "data leakage," where models achieve high performance by memorizing training examples rather than learning generalizable patterns. This paper demonstrates that reported performance metrics can be inflated by up to 100% and introduces a toolset for data deduplication to ensure more realistic evaluations.

Introduction

The field of "Big Code" leverages massive repositories like GitHub to train models for tasks such as code completion, variable naming, and bug detection. However, GitHub is characterized by high levels of duplication due to forks, library bundling, and code reuse. When similar code appears in both training and test sets, the model performs a lookup rather than learning, violating the fundamental machine learning assumption that data points are independent and identically distributed (i.i.d.). This "looming crisis" leads researchers to overestimate the real-world utility of their tools for developers writing original code.

Methods and Taxonomy

To address this, the author re-implemented SourcererCC, a token-based duplicate detection tool. Instead of exact matches, it uses Jaccard similarity thresholds (e.g., 80% for identifiers) to identify "near-duplicates" that differ only in comments or formatting.

The paper establishes a taxonomy of three critical duplicate types:

  • In-train: Duplicates within the training set, causing models to overweight specific samples.
  • In-test: Duplicates within the test set, which skew final metrics.
  • Cross-set: The most detrimental type, where code appears in both training and testing sets, leading to artificial performance inflation.
Taxonomy of duplicates in training and test sets.

Datasets and Duplication Extent

The study analyzed ten popular "Big Code" datasets, including Java-Large, JavaScript-150k, and Python-150k. A striking finding was that the Concode dataset contained approximately 68.7% duplicate methods. Even datasets with prior deduplication efforts still harbored significant "near-duplicates," highlighting the inadequacy of simple exact-match filtering.

Key Results and Insights

The impact of removing cross-set duplicates is profound across various tasks and architectures. The "Unbiased" metric—evaluating on non-duplicated data—is the only true reflection of a human developer's experience.

Performance drops across various tasks after deduplication.

Key findings include:

  • Performance Inflation: Accuracy for predicting identifiers in JavaScript code completion dropped by 51.4% after deduplication. Variable naming accuracy (JsNice) saw a 37.4% relative decrease, while method naming (code2vec) F1 scores dropped by 12.3%.
  • The Capacity Trap: Larger models with more parameters are more susceptible to duplication bias. Their increased "memory" allows them to store duplicated snippets, achieving high scores through memorization rather than generalization.
  • Identifier Sparsity: Duplication makes rare identifiers appear more frequent and predictable than they are in novel code, further inflating metrics.

Conclusion and Future Work

The study reveals that many "state-of-the-art" results are significantly inflated by duplication bias. While this research focused on file and method-level duplication, smaller "clones" or common idioms remain a challenge. Future work should explore temporal splitting (training on old code, testing on new) and the development of architectures inherently robust to duplication, such as those that penalize memorization or utilize retrieval-augmented approaches.

Summary updated

Abstract

The field of big code relies on mining large corpora of code to perform some learning task. A significant threat to this approach has been recently identified by Lopes et al. (2017) who found a large amount of near-duplicate code on GitHub. However, the impact of code duplication has not been noticed by researchers devising machine learning models for source code. In this work, we explore the effects of code duplication on machine learning models showing that reported performance metrics are sometimes inflated by up to 100% when testing on duplicated code corpora compared to the performance on de-duplicated corpora which more accurately represent how machine learning models of code are used by software engineers. We present a duplication index for widely used datasets, list best practices for collecting code corpora and evaluating machine learning models on them. Finally, we release tools to help the community avoid this problem in future research.
BibTeX
@inproceedings{allamanis2018adverse,
  author = {Miltiadis Allamanis},
  title = {{The Adverse Effects of Code Duplication in Machine Learning Models of Code}},
  booktitle = {Onward!},
  year = {2018},
  doi = {10.1145/3359591.3359735},
}
Figure