ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning

Shangqing Liu, Bozhi Wu, Xiaofei Xie, Guozhu Meng, Yang Liu

Abstract

Pre-trained models for code, such as CodeBERT and GraphCodeBERT, have significantly advanced software engineering tasks. However, these models often exhibit a "glass jaw"—their performance can plummet when faced with simple, semantic-preserving syntactic changes like variable renaming. We present ContraBERT, a framework that integrates contrastive learning with a suite of nine data augmentation techniques. By teaching models that code meaning remains invariant across syntactic transformations, ContraBERT substantially improves both robustness and accuracy across various downstream tasks, including clone detection and defect identification.

Introduction

Large Language Models for Code (LLMC) have revolutionized tasks like bug detection and code translation. Despite their success, these models are highly susceptible to adversarial attacks. A primary reason is that standard pre-training objectives, such as Masked Language Modeling (MLM), focus on local token patterns rather than global semantics. For instance, renaming a variable from counter to temp_val can cause a model's performance in clone detection to drop significantly. This fragility poses a major hurdle for deploying these models in security-critical or production environments.

Methodology

ContraBERT addresses this semantic gap by adding a contrastive learning objective to the standard pre-training process. The framework relies on two core components: data augmentation and a momentum-based contrastive loss.

Data Augmentation

To teach the model what "equivalent" code looks like, the authors designed nine operators across two domains:

  • Programming Language (PL): Renaming functions, renaming variables, inserting dead code, reordering independent statements, and random line sampling.
  • Natural Language (NL): Back-translation (e.g., English to German and back), and random word deletion, switching, or copying.

Contrastive Pre-training

Instead of solely predicting missing tokens, ContraBERT is trained to recognize that an original code snippet and its augmented variant are "positive pairs" (semantically identical) while pushing them away from unrelated "negative pairs" in the embedding space.

The ContraBERT pre-training architecture combines Masked Language Modeling with an InfoNCE contrastive loss using a momentum encoder (M').

The architecture utilizes a Momentum Contrast (MoCo) approach. This involves a main encoder ($M$) that learns via gradient descent and a momentum encoder ($M'$) that maintains a queue of dissimilar samples. This allows the model to efficiently compare a snippet against thousands of negative examples without requiring massive batch sizes.

Experimental Results

ContraBERT was pre-trained on the CodeSearchNet dataset (2.1 million code-comment pairs) and evaluated on benchmarks including POJ-104 (clone detection) and Devign (defect detection).

Robustness and Adversarial Attacks

The most critical finding was the model's resilience to variable renaming. In adversarial tests on clone detection, as the number of renamed variables increased, the performance of baseline models like CodeBERT decayed rapidly. ContraBERT maintained a significantly higher and smoother performance curve.

Accuracy vs. the number of renaming edits: ContraBERT variants maintain significantly higher robustness compared to CodeBERT and GraphCodeBERT as adversarial edits increase.

Downstream Performance

  • Clone Detection: On the POJ-104 benchmark, ContraBERT achieved a 90.46 MAP@R, a substantial improvement over CodeBERT’s 84.29.
  • Generalization: Even on non-adversarial benchmarks, ContraBERT outperformed baselines in defect detection and code translation, suggesting that improved robustness correlates with better overall semantic understanding.

Discussion and Key Insights

Analysis revealed that the "Rename Variable" (RV) operator is the most impactful augmentation. Removing RV from training caused the largest drop in robustness, confirming that models naturally over-rely on specific variable names. Furthermore, visualization of embeddings via T-SNE plots showed that ContraBERT forms much tighter, distinct clusters for semantically similar code compared to the scattered embeddings of baseline models.

Conclusion and Future Work

ContraBERT demonstrates that contrastive learning is a powerful tool for hardening code models against syntactic noise. While the current framework shows significant gains, future work could explore more complex, compiler-level transformations—such as loop unrolling or control flow restructuring—to further bridge the semantic gap between natural language queries and raw source code.

Summary updated

BibTeX
@inproceedings{liu2023contrabert,
  author = {Shangqing Liu and Bozhi Wu and Xiaofei Xie and Guozhu Meng and Yang Liu},
  title = {{ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning}},
  booktitle = {ICSE},
  year = {2023},
  url = {https://openreview.net/forum?id=6GsV1ZwKk6d},
}
Figure