Abstract
We propose Corder, a self-supervised contrastive learning framework for source code model. Corder is designed to alleviate the need of labeled data for code retrieval and code summarization tasks. The pre-trained model of Corder can be used in two ways: (1) it can produce vector representation of code which can be applied to code retrieval tasks that do not have labeled data; (2) it can be used in a fine-tuning process for tasks that might still require label data such as code summarization. The key innovation is that we train the source code model by asking it to recognize similar and dissimilar code snippets through a contrastive learning objective. To do so, we use a set of semantic-preserving transformation operators to generate code snippets that are syntactically diverse but semantically equivalent. Through extensive experiments, we have shown that the code models pretrained by Corder substantially outperform the other baselines for code-to-code retrieval, text-to-code retrieval, and code-to-text summarization tasks.
TL;DR
Corder is a self-supervised framework that learns powerful code representations by training a model to recognize when two syntactically different snippets of code actually do the same thing. By applying semantic-preserving transformations (like swapping independent statements or exchanging loops), Corder eliminates the need for massive labeled datasets, significantly boosting performance in code retrieval and summarization tasks.
Introduction: The Challenge of Labeled Code Data
Deep learning has revolutionized how we search for, summarize, and repair source code. However, these models are "data-hungry," typically requiring vast amounts of human-labeled data, such as code paired with natural language descriptions.
Existing self-supervised methods often rely on noisy heuristics, like assuming a method's name or its accompanying comment perfectly describes its logic. However, generic names like do_stuff() are common and unhelpful, leading models to learn incorrect patterns. Corder addresses this by asking: Can we teach a model the "essence" of code without any human labels at all?
Methods: Contrastive Learning with Corder
The authors introduce Corder, which utilizes Contrastive Learning—a technique where a model learns to pull "similar" items together in a vector space and push "dissimilar" items apart.
Semantic-Preserving Transformations
The core innovation lies in how similarity is defined. Instead of relying on external labels, Corder uses five semantic-preserving transformation operators to create variants of the same code snippet:
- Variable Renaming (VN): Changing variable names to random tokens.
- Unused Statement (US): Inserting "dead code" that doesn't affect the output.
- Permutation of Statements (PS): Swapping two independent statements.
- Loop Exchange (LX): Converting a
forloop to awhileloop (or vice versa). - Switch to If (SF): Replacing a
switchstatement with an equivalentif-elsechain.
The Corder Workflow
The training process follows a specific pipeline:
- Augmentation: Take an unlabeled code snippet and apply two different transformations to create two "augmented" versions.
- Encoding: Feed these into a neural encoder (such as a Transformer, TBCNN, Code2vec, or LSTM).
- Contrastive Loss: Use a Noise Contrastive Estimate (NCE) loss to train the encoder to recognize that these two versions are semantically identical, while all other snippets in the batch are treated as dissimilar.
Datasets
The authors used a mix of large-scale unlabeled data for pre-training and specialized datasets for evaluation:
- Pre-training: JavaSmall (700k examples) and JavaMed (4M examples from 1,000 GitHub projects).
- Code-to-Code Retrieval: The OJ dataset (52,000 C programs) and BigCloneBench (6 million clone pairs).
- Text-to-Code & Summarization: The DeepCS dataset (16 million Java methods with docstrings).
Key Insights
- Structure Over Syntax: Ablation studies revealed that transformations changing the code's structure (like Permutation of Statements or Switch-to-If) are far more effective for learning than simple Variable Renaming. This suggests models must "see" through structural noise to understand logic.
- Pre-training Scale: Models pre-trained on the larger
JavaMeddataset consistently outperformed those onJavaSmall, demonstrating that the framework scales effectively with more unlabeled data. - Universal Applicability: Corder is architecture-agnostic, showing improvements across various encoders including TBCNN, Code2vec, LSTMs, and Transformers.
- Superior to Supervised Learning: In code classification tasks, Corder-pretrained embeddings outperformed embeddings from models trained specifically for those tasks (like AdaSent), indicating that contrastive learning captures more robust features.
Results
Corder demonstrated state-of-the-art performance across three major benchmarks:
- Code-to-Code Retrieval: On BigCloneBench, Corder achieved an MRR of 0.832, significantly outperforming specialized baselines like FaCoy (0.514).
- Text-to-Code Retrieval: Fine-tuning a Corder-pretrained Transformer reached a Precision@10 of 0.881, compared to 0.793 for a Transformer trained from scratch.
- Code Summarization: Corder improved the BLEU score for code2seq from 22.89 to 26.56, a substantial jump in the quality of generated comments.
Limitations and Future Work
While effective, Corder has some constraints. Certain transformations, like "Switch to If," can only be applied if the original code contains a switch statement, limiting the diversity of positive pairs for some snippets. Additionally, while the system uses srcML for multi-language support, the transformations themselves require language-specific logic. Future research will explore more complex transformations and the application of this framework to vulnerability detection.
Summary updated