Self-Supervised Contrastive Learning with Adversarial Perturbations for Robust Pretrained Language Models

Zhao Meng, Yihan Dong, Mrinmaya Sachan, Roger Wattenhofer

Introduction

Pretrained language models (PLMs) like BERT have revolutionized NLP but remain notoriously vulnerable to word-substitution adversarial attacks. In these scenarios, replacing just a few words with synonyms can flip a model's prediction, such as changing "good" to "great" to deceive a sentiment classifier. While adversarial training is the standard defense, it is computationally expensive and typically requires large amounts of labeled data, limiting its application to specific downstream tasks.

This paper introduces Adversarial Contrastive Learning (ADCL), a self-supervised approach that builds robustness into foundational models before fine-tuning. By generating "hard positive" adversarial examples on the fly and pulling their representations closer to clean text in the vector space, ADCL significantly hardens models against attacks without requiring any labeled data.

Adversarial Contrastive Learning (ADCL)

The core of the ADCL method is a self-supervised contrastive learning framework. The model is trained to pull the representations of a text and its augmented version (a positive pair) together while pushing different texts (negative pairs) apart.

Overview of the Adversarial Contrastive Learning (ADCL) framework, illustrating (a) the generation of adversarial perturbations, (b) the contrastive pull/push mechanism in the embedding space, and (c) the resulting robust clusters.

The innovation lies in the creation of the positive pair. Instead of simple augmentations like back-translation, the authors introduce a Geometry Attack. This attack utilizes the gradients of the contrastive loss to identify words that, if substituted, would maximize the loss. By iteratively substituting these words, the system generates challenging adversarial examples that force the model to learn representations invariant to such perturbations.

Experimental Results

The authors evaluated ADCL on four standard text classification benchmarks: AG's News, Yelp, IMDB, and DBpedia. Robustness was measured against several attack methods, including Geometry, TextFooler, PWWS, and BAE-R.

Performance and robustness metrics across four benchmark datasets, comparing standard pretraining (NP), basic contrastive learning (BTCL), and the proposed ADCL.

Key Findings

  • Improved Baseline Robustness: On the AG's News dataset, a standard BERT model (NP) has an 86.2% success rate for the Geometry Attack. With ADCL pretraining, this drops to 76.5% even before adversarial fine-tuning.
  • Synergy with Supervised Defense: ADCL acts as a multiplier for existing defenses. When combined with supervised adversarial fine-tuning (ADV), the attack success rate on AG's News plummets to 18.7%, outperforming standard adversarial training (20.7%).
  • Cross-Domain Generalization: Because ADCL is unsupervised, it can be pretrained on out-of-domain data. Pretraining on DBpedia and fine-tuning on IMDB reduced the TextFooler attack success rate from 98.7% to 92.3%.
  • Efficiency: The Geometry Attack is highly optimized, running at least 4 times faster than TextFooler and up to 10 times faster than PWWS, making on-the-fly adversarial generation feasible during pretraining.

Discussion and Insights

The success of ADCL demonstrates that labels are not strictly necessary for building model robustness. By grouping clean and adversarial examples based on semantic similarity, the model learns a more stable embedding space. Vector space analysis confirms that contrastive pretraining physically pulls clean and adversarial embeddings closer together, making it harder for attackers to push a representation across a decision boundary.

Conclusion and Future Work

ADCL provides a scalable path toward robust foundational models. While current experiments were limited by computational infrastructure to datasets like DBpedia, the authors plan to scale this method to massive web corpora like Wikipedia. This would determine if robustness gains scale alongside general performance, potentially leading to inherently more secure language models out of the box.

Summary updated

BibTeX
@inproceedings{meng2022self,
  author = {Zhao Meng and Yihan Dong and Mrinmaya Sachan and Roger Wattenhofer},
  title = {{Self-Supervised Contrastive Learning with Adversarial Perturbations for Robust Pretrained Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.8},
}
Figure