Abstract
Pre-trained models for code, such as CodeBERT and GraphCodeBERT, have significantly advanced software engineering tasks. However, these models often exhibit a "glass jaw"—their performance can plummet when faced with simple, semantic-preserving syntactic changes like variable renaming. We present ContraBERT, a framework that integrates contrastive learning with a suite of nine data augmentation techniques. By teaching models that code meaning remains invariant across syntactic transformations, ContraBERT substantially improves both robustness and accuracy across various downstream tasks, including clone detection and defect identification.
Introduction
Large Language Models for Code (LLMC) have revolutionized tasks like bug detection and code translation. Despite their success, these models are highly susceptible to adversarial attacks. A primary reason is that standard pre-training objectives, such as Masked Language Modeling (MLM), focus on local token patterns rather than global semantics. For instance, renaming a variable from counter to temp_val can cause a model's performance in clone detection to drop significantly. This fragility poses a major hurdle for deploying these models in security-critical or production environments.
Methodology
ContraBERT addresses this semantic gap by adding a contrastive learning objective to the standard pre-training process. The framework relies on two core components: data augmentation and a momentum-based contrastive loss.
Data Augmentation
To teach the model what "equivalent" code looks like, the authors designed nine operators across two domains:
- Programming Language (PL): Renaming functions, renaming variables, inserting dead code, reordering independent statements, and random line sampling.
- Natural Language (NL): Back-translation (e.g., English to German and back), and random word deletion, switching, or copying.
Contrastive Pre-training
Instead of solely predicting missing tokens, ContraBERT is trained to recognize that an original code snippet and its augmented variant are "positive pairs" (semantically identical) while pushing them away from unrelated "negative pairs" in the embedding space.
The architecture utilizes a Momentum Contrast (MoCo) approach. This involves a main encoder ($M$) that learns via gradient descent and a momentum encoder ($M'$) that maintains a queue of dissimilar samples. This allows the model to efficiently compare a snippet against thousands of negative examples without requiring massive batch sizes.
Experimental Results
ContraBERT was pre-trained on the CodeSearchNet dataset (2.1 million code-comment pairs) and evaluated on benchmarks including POJ-104 (clone detection) and Devign (defect detection).
Robustness and Adversarial Attacks
The most critical finding was the model's resilience to variable renaming. In adversarial tests on clone detection, as the number of renamed variables increased, the performance of baseline models like CodeBERT decayed rapidly. ContraBERT maintained a significantly higher and smoother performance curve.
Downstream Performance
- Clone Detection: On the POJ-104 benchmark, ContraBERT achieved a 90.46 MAP@R, a substantial improvement over CodeBERT’s 84.29.
- Generalization: Even on non-adversarial benchmarks, ContraBERT outperformed baselines in defect detection and code translation, suggesting that improved robustness correlates with better overall semantic understanding.
Discussion and Key Insights
Analysis revealed that the "Rename Variable" (RV) operator is the most impactful augmentation. Removing RV from training caused the largest drop in robustness, confirming that models naturally over-rely on specific variable names. Furthermore, visualization of embeddings via T-SNE plots showed that ContraBERT forms much tighter, distinct clusters for semantically similar code compared to the scattered embeddings of baseline models.
Conclusion and Future Work
ContraBERT demonstrates that contrastive learning is a powerful tool for hardening code models against syntactic noise. While the current framework shows significant gains, future work could explore more complex, compiler-level transformations—such as loop unrolling or control flow restructuring—to further bridge the semantic gap between natural language queries and raw source code.
Summary updated