Abstract
Machine learning models for code are frequently evaluated on datasets containing significant amounts of near-duplicate files. This leads to "data leakage," where models achieve high performance by memorizing training examples rather than learning generalizable patterns. This paper demonstrates that reported performance metrics can be inflated by up to 100% and introduces a toolset for data deduplication to ensure more realistic evaluations.
Introduction
The field of "Big Code" leverages massive repositories like GitHub to train models for tasks such as code completion, variable naming, and bug detection. However, GitHub is characterized by high levels of duplication due to forks, library bundling, and code reuse. When similar code appears in both training and test sets, the model performs a lookup rather than learning, violating the fundamental machine learning assumption that data points are independent and identically distributed (i.i.d.). This "looming crisis" leads researchers to overestimate the real-world utility of their tools for developers writing original code.
Methods and Taxonomy
To address this, the author re-implemented SourcererCC, a token-based duplicate detection tool. Instead of exact matches, it uses Jaccard similarity thresholds (e.g., 80% for identifiers) to identify "near-duplicates" that differ only in comments or formatting.
The paper establishes a taxonomy of three critical duplicate types:
- In-train: Duplicates within the training set, causing models to overweight specific samples.
- In-test: Duplicates within the test set, which skew final metrics.
- Cross-set: The most detrimental type, where code appears in both training and testing sets, leading to artificial performance inflation.
Datasets and Duplication Extent
The study analyzed ten popular "Big Code" datasets, including Java-Large, JavaScript-150k, and Python-150k. A striking finding was that the Concode dataset contained approximately 68.7% duplicate methods. Even datasets with prior deduplication efforts still harbored significant "near-duplicates," highlighting the inadequacy of simple exact-match filtering.
Key Results and Insights
The impact of removing cross-set duplicates is profound across various tasks and architectures. The "Unbiased" metric—evaluating on non-duplicated data—is the only true reflection of a human developer's experience.
Key findings include:
- Performance Inflation: Accuracy for predicting identifiers in JavaScript code completion dropped by 51.4% after deduplication. Variable naming accuracy (JsNice) saw a 37.4% relative decrease, while method naming (code2vec) F1 scores dropped by 12.3%.
- The Capacity Trap: Larger models with more parameters are more susceptible to duplication bias. Their increased "memory" allows them to store duplicated snippets, achieving high scores through memorization rather than generalization.
- Identifier Sparsity: Duplication makes rare identifiers appear more frequent and predictable than they are in novel code, further inflating metrics.
Conclusion and Future Work
The study reveals that many "state-of-the-art" results are significantly inflated by duplication bias. While this research focused on file and method-level duplication, smaller "clones" or common idioms remain a challenge. Future work should explore temporal splitting (training on old code, testing on new) and the development of architectures inherently robust to duplication, such as those that penalize memorization or utilize retrieval-augmented approaches.
Summary updated