Abstract
Over the past few years, deep learning (DL) has become a driving force for large-scale source code analysis. However, distribution shift—where test data follows a different distribution from the training set—poses a significant challenge for the reliable deployment of these models due to unexpected accuracy degradation. While progress on distribution shift benchmarking has been made in computer vision and natural language processing, limited progress has been made for source code tasks. This paper introduces CodeS, a comprehensive distribution shift benchmark dataset for source code learning. CodeS supports two programming languages (Java and Python) and five types of code distribution shifts: task, programmer, time-stamp, token, and CST. Our evaluation of popular code learning models and out-of-distribution (OOD) detectors reveals that current models are fragile under representation-based shifts, though pre-trained models exhibit higher resistance. CodeS is publicly available to facilitate research on the quality assessment of code learning models.
Introduction
Deep learning has revolutionized source code analysis, enabling tasks like auto-completion, bug detection, and code summarization. Despite these advances, most models are evaluated on "in-distribution" (ID) data, where test sets mirror training sets. In real-world scenarios, code is dynamic: syntax evolves over time, and different developers possess unique coding styles. This phenomenon, known as distribution shift, often leads to unexpected performance drops when models are deployed.
Until now, there has been a lack of fine-grained benchmarks to measure how these shifts impact model performance or to evaluate the effectiveness of OOD detection tools in the context of source code. CodeS fills this gap by providing a systematic framework for analyzing both natural and representation-based shifts.
The CodeS Benchmark
The researchers developed a systematic way to define and generate distribution shifts for code, categorizing them into two perspectives:
- Natural Distribution Shifts: These mimic real-world scenarios, including Cross-task (different programming problems), Cross-programmer (different developers), and Time-period (code written at different times).
- Representation-based Shifts: These measure structural and lexical differences, specifically Token Frequency (shifts in keywords/operators) and CST Similarity (shifts in the Concrete Syntax Tree structure).
As shown in Figure 1, CodeS is the first benchmark to comprehensively cover all five types of shifts, providing a more robust testing ground than previous datasets like Wilds. The benchmark includes datasets such as Java250-S, Python800-S (derived from Project CodeNet), and Python75 (crawled from AtCoder).
Experimental Evaluation
The study evaluated five deep learning architectures—ranging from standard CNNs and MLPs to state-of-the-art pre-trained models like CodeBERT and GraphCodeBERT—against four popular OOD detection algorithms.
Key Insights
- Fragility of Standard Models: Representation-based shifts (Token and CST) caused more significant performance drops than natural shifts. For example, on the Python75 dataset, a CNN's accuracy dropped from 96.91% (ID) to 61.04% under a token frequency shift.
- The Robustness of Pre-training: Bimodal pre-trained models are significantly more robust. As illustrated in Figure 4, while basic models struggle, models like GraphCodeBERT maintain high accuracy even under significant shifts, such as the token shift in Python800-S where it achieved 97.77% accuracy.
- OOD Detection Challenges: Standard OOD detectors from computer vision and NLP performed poorly on code. Interestingly, the simple Maximum Softmax Probability (MSP) method often outperformed more complex alternatives, suggesting that the structured nature of code requires specialized detection techniques.
Conclusion
CodeS highlights a sobering reality: current code models are highly fragile when faced with real-world distribution shifts. While pre-trained models offer better generalization, there is still a massive gap in our ability to monitor and detect when these models are "out of their depth." Future work will extend CodeS to more complex tasks like bug detection and incorporate industrial code from platforms like GitHub to capture a broader range of professional coding styles.
Summary updated