Efficient Framework for Learning Code Representations through Semantic-Preserving Program Transformations

Nghi D. Q. Bui

Abstract

We propose Corder, a self-supervised contrastive learning framework for source code model. Corder is designed to alleviate the need of labeled data for code retrieval and code summarization tasks. The pre-trained model of Corder can be used in two ways: (1) it can produce vector representation of code which can be applied to code retrieval tasks that do not have labeled data; (2) it can be used in a fine-tuning process for tasks that might still require label data such as code summarization. The key innovation is that we train the source code model by asking it to recognize similar and dissimilar code snippets through a contrastive learning objective. To do so, we use a set of semantic-preserving transformation operators to generate code snippets that are syntactically diverse but semantically equivalent. Through extensive experiments, we have shown that the code models pretrained by Corder substantially outperform the other baselines for code-to-code retrieval, text-to-code retrieval, and code-to-text summarization tasks.

TL;DR

Corder is a self-supervised framework that learns powerful code representations by training a model to recognize when two syntactically different snippets of code actually do the same thing. By applying semantic-preserving transformations (like swapping independent statements or exchanging loops), Corder eliminates the need for massive labeled datasets, significantly boosting performance in code retrieval and summarization tasks.

Introduction: The Challenge of Labeled Code Data

Deep learning has revolutionized how we search for, summarize, and repair source code. However, these models are "data-hungry," typically requiring vast amounts of human-labeled data, such as code paired with natural language descriptions.

Existing self-supervised methods often rely on noisy heuristics, like assuming a method's name or its accompanying comment perfectly describes its logic. However, generic names like do_stuff() are common and unhelpful, leading models to learn incorrect patterns. Corder addresses this by asking: Can we teach a model the "essence" of code without any human labels at all?

Methods: Contrastive Learning with Corder

The authors introduce Corder, which utilizes Contrastive Learning—a technique where a model learns to pull "similar" items together in a vector space and push "dissimilar" items apart.

Semantic-Preserving Transformations

The core innovation lies in how similarity is defined. Instead of relying on external labels, Corder uses five semantic-preserving transformation operators to create variants of the same code snippet:

  1. Variable Renaming (VN): Changing variable names to random tokens.
  2. Unused Statement (US): Inserting "dead code" that doesn't affect the output.
  3. Permutation of Statements (PS): Swapping two independent statements.
  4. Loop Exchange (LX): Converting a for loop to a while loop (or vice versa).
  5. Switch to If (SF): Replacing a switch statement with an equivalent if-else chain.
Example of semantic-preserving transformations: (a) original code, (b) variable renaming, and (c) statement swapping.

The Corder Workflow

The training process follows a specific pipeline:

  • Augmentation: Take an unlabeled code snippet and apply two different transformations to create two "augmented" versions.
  • Encoding: Feed these into a neural encoder (such as a Transformer, TBCNN, Code2vec, or LSTM).
  • Contrastive Loss: Use a Noise Contrastive Estimate (NCE) loss to train the encoder to recognize that these two versions are semantically identical, while all other snippets in the batch are treated as dissimilar.
The Corder framework: transforming unlabeled code snippets and using a neural encoder with a contrastive loss to learn representations.

Datasets

The authors used a mix of large-scale unlabeled data for pre-training and specialized datasets for evaluation:

  • Pre-training: JavaSmall (700k examples) and JavaMed (4M examples from 1,000 GitHub projects).
  • Code-to-Code Retrieval: The OJ dataset (52,000 C programs) and BigCloneBench (6 million clone pairs).
  • Text-to-Code & Summarization: The DeepCS dataset (16 million Java methods with docstrings).

Key Insights

  • Structure Over Syntax: Ablation studies revealed that transformations changing the code's structure (like Permutation of Statements or Switch-to-If) are far more effective for learning than simple Variable Renaming. This suggests models must "see" through structural noise to understand logic.
  • Pre-training Scale: Models pre-trained on the larger JavaMed dataset consistently outperformed those on JavaSmall, demonstrating that the framework scales effectively with more unlabeled data.
  • Universal Applicability: Corder is architecture-agnostic, showing improvements across various encoders including TBCNN, Code2vec, LSTMs, and Transformers.
  • Superior to Supervised Learning: In code classification tasks, Corder-pretrained embeddings outperformed embeddings from models trained specifically for those tasks (like AdaSent), indicating that contrastive learning captures more robust features.

Results

Corder demonstrated state-of-the-art performance across three major benchmarks:

  • Code-to-Code Retrieval: On BigCloneBench, Corder achieved an MRR of 0.832, significantly outperforming specialized baselines like FaCoy (0.514).
  • Text-to-Code Retrieval: Fine-tuning a Corder-pretrained Transformer reached a Precision@10 of 0.881, compared to 0.793 for a Transformer trained from scratch.
  • Code Summarization: Corder improved the BLEU score for code2seq from 22.89 to 26.56, a substantial jump in the quality of generated comments.

Limitations and Future Work

While effective, Corder has some constraints. Certain transformations, like "Switch to If," can only be applied if the original code contains a switch statement, limiting the diversity of positive pairs for some snippets. Additionally, while the system uses srcML for multi-language support, the transformations themselves require language-specific logic. Future research will explore more complex transformations and the application of this framework to vulnerability detection.

Summary updated

Abstract

We propose Corder, a self-supervised contrastive learning framework for source code model. Corder is designed to alleviate the need of labeled data for code retrieval and code summarization tasks. The pre-trained model of Corder can be used in two ways: (1) it can produce vector representation of code which can be applied to code retrieval tasks that do not have labeled data; (2) it can be used in a fine-tuning process for tasks that might still require label data such as code summarization. The key innovation is that we train the source code model by asking it to recognize similar and dissimilar code snippets through a contrastive learning objective. To do so, we use a set of semantic-preserving transformation operators to generate code snippets that are syntactically diverse but semantically equivalent. Through extensive experiments, we have shown that the code models pretrained by Corder substantially outperform the other baselines for code-to-code retrieval, text-to-code retrieval, and code-to-text summarization tasks.
BibTeX
@inproceedings{bui2021efficient,
  author = {Nghi D. Q. Bui},
  title = {{Efficient Framework for Learning Code Representations through Semantic-Preserving Program Transformations}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2021},
  url = {https://openreview.net/forum?id=bXtcbWnp7B},
}
Figure