CodeS: A Distribution Shift Benchmark Dataset for Source Code Learning

Qiang Hu, Yuejun Guo, Xiaofei Xie, Maxime Cordy, Lei Ma, Mike Papadakis, Yves Le TraonarXiv 2022 · arXiv · OpenReview

Abstract

Over the past few years, deep learning (DL) has become a driving force for large-scale source code analysis. However, distribution shift—where test data follows a different distribution from the training set—poses a significant challenge for the reliable deployment of these models due to unexpected accuracy degradation. While progress on distribution shift benchmarking has been made in computer vision and natural language processing, limited progress has been made for source code tasks. This paper introduces CodeS, a comprehensive distribution shift benchmark dataset for source code learning. CodeS supports two programming languages (Java and Python) and five types of code distribution shifts: task, programmer, time-stamp, token, and CST. Our evaluation of popular code learning models and out-of-distribution (OOD) detectors reveals that current models are fragile under representation-based shifts, though pre-trained models exhibit higher resistance. CodeS is publicly available to facilitate research on the quality assessment of code learning models.

Introduction

Deep learning has revolutionized source code analysis, enabling tasks like auto-completion, bug detection, and code summarization. Despite these advances, most models are evaluated on "in-distribution" (ID) data, where test sets mirror training sets. In real-world scenarios, code is dynamic: syntax evolves over time, and different developers possess unique coding styles. This phenomenon, known as distribution shift, often leads to unexpected performance drops when models are deployed.

Until now, there has been a lack of fine-grained benchmarks to measure how these shifts impact model performance or to evaluate the effectiveness of OOD detection tools in the context of source code. CodeS fills this gap by providing a systematic framework for analyzing both natural and representation-based shifts.

The CodeS Benchmark

The researchers developed a systematic way to define and generate distribution shifts for code, categorizing them into two perspectives:

  1. Natural Distribution Shifts: These mimic real-world scenarios, including Cross-task (different programming problems), Cross-programmer (different developers), and Time-period (code written at different times).
  2. Representation-based Shifts: These measure structural and lexical differences, specifically Token Frequency (shifts in keywords/operators) and CST Similarity (shifts in the Concrete Syntax Tree structure).
Comparison of CodeS with existing benchmarks across various shift types.

As shown in Figure 1, CodeS is the first benchmark to comprehensively cover all five types of shifts, providing a more robust testing ground than previous datasets like Wilds. The benchmark includes datasets such as Java250-S, Python800-S (derived from Project CodeNet), and Python75 (crawled from AtCoder).

Experimental Evaluation

The study evaluated five deep learning architectures—ranging from standard CNNs and MLPs to state-of-the-art pre-trained models like CodeBERT and GraphCodeBERT—against four popular OOD detection algorithms.

Key Insights

  • Fragility of Standard Models: Representation-based shifts (Token and CST) caused more significant performance drops than natural shifts. For example, on the Python75 dataset, a CNN's accuracy dropped from 96.91% (ID) to 61.04% under a token frequency shift.
  • The Robustness of Pre-training: Bimodal pre-trained models are significantly more robust. As illustrated in Figure 4, while basic models struggle, models like GraphCodeBERT maintain high accuracy even under significant shifts, such as the token shift in Python800-S where it achieved 97.77% accuracy.
  • OOD Detection Challenges: Standard OOD detectors from computer vision and NLP performed poorly on code. Interestingly, the simple Maximum Softmax Probability (MSP) method often outperformed more complex alternatives, suggesting that the structured nature of code requires specialized detection techniques.
Model generalization results (ID vs OOD test accuracy) for different architectures across shift types.

Conclusion

CodeS highlights a sobering reality: current code models are highly fragile when faced with real-world distribution shifts. While pre-trained models offer better generalization, there is still a massive gap in our ability to monitor and detect when these models are "out of their depth." Future work will extend CodeS to more complex tasks like bug detection and incorporate industrial code from platforms like GitHub to capture a broader range of professional coding styles.

Summary updated

Abstract

Over the past few years, deep learning (DL) has been continuously expanding its applications and becoming a driving force for large-scale source code analysis in the big code era. Distribution shift, where the test set follows a different distribution from the training set, has been a longstanding challenge for the reliable deployment of DL models due to the unexpected accuracy degradation. Although recent progress on distribution shift benchmarking has been made in domains such as computer vision and natural language process. Limited progress has been made on distribution shift analysis and benchmarking for source code tasks, on which there comes a strong demand due to both its volume and its important role in supporting the foundations of almost all industrial sectors. To fill this gap, this paper initiates to propose CodeS, a distribution shift benchmark dataset, for source code learning. Specifically, CodeS supports 2 programming languages (i.e., Java and Python) and 5 types of code distribution shifts (i.e., task, programmer, time-stamp, token, and CST). To the best of our knowledge, we are the first to define the code representation-based distribution shifts. In the experiments, we first evaluate the effectiveness of existing out-of-distribution detectors and the reasonability of the distribution shift definitions and then measure the model generalization of popular code learning models (e.g., CodeBERT) on classification task. The results demonstrate that 1) only softmax score-based OOD detectors perform well on CodeS, 2) distribution shift causes the accuracy degradation in all code classification models, 3) representation-based distribution shifts have a higher impact on the model than others, and 4) pre-trained models are more resistant to distribution shifts. We make CodeS publicly available, enabling follow-up research on the quality assessment of code learning models.
BibTeX
@misc{hu2022codes,
  author = {Qiang Hu and Yuejun Guo and Xiaofei Xie and Maxime Cordy and Lei Ma and Mike Papadakis and Yves Le Traon},
  title = {{CodeS: A Distribution Shift Benchmark Dataset for Source Code Learning}},
  year = {2022},
  eprint = {2206.05480},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.05480},
}

Loading related content...

Figure