Introduction
Pretrained language models (PLMs) like BERT have revolutionized NLP but remain notoriously vulnerable to word-substitution adversarial attacks. In these scenarios, replacing just a few words with synonyms can flip a model's prediction, such as changing "good" to "great" to deceive a sentiment classifier. While adversarial training is the standard defense, it is computationally expensive and typically requires large amounts of labeled data, limiting its application to specific downstream tasks.
This paper introduces Adversarial Contrastive Learning (ADCL), a self-supervised approach that builds robustness into foundational models before fine-tuning. By generating "hard positive" adversarial examples on the fly and pulling their representations closer to clean text in the vector space, ADCL significantly hardens models against attacks without requiring any labeled data.
Adversarial Contrastive Learning (ADCL)
The core of the ADCL method is a self-supervised contrastive learning framework. The model is trained to pull the representations of a text and its augmented version (a positive pair) together while pushing different texts (negative pairs) apart.
The innovation lies in the creation of the positive pair. Instead of simple augmentations like back-translation, the authors introduce a Geometry Attack. This attack utilizes the gradients of the contrastive loss to identify words that, if substituted, would maximize the loss. By iteratively substituting these words, the system generates challenging adversarial examples that force the model to learn representations invariant to such perturbations.
Experimental Results
The authors evaluated ADCL on four standard text classification benchmarks: AG's News, Yelp, IMDB, and DBpedia. Robustness was measured against several attack methods, including Geometry, TextFooler, PWWS, and BAE-R.
Key Findings
- Improved Baseline Robustness: On the AG's News dataset, a standard BERT model (NP) has an 86.2% success rate for the Geometry Attack. With ADCL pretraining, this drops to 76.5% even before adversarial fine-tuning.
- Synergy with Supervised Defense: ADCL acts as a multiplier for existing defenses. When combined with supervised adversarial fine-tuning (ADV), the attack success rate on AG's News plummets to 18.7%, outperforming standard adversarial training (20.7%).
- Cross-Domain Generalization: Because ADCL is unsupervised, it can be pretrained on out-of-domain data. Pretraining on DBpedia and fine-tuning on IMDB reduced the TextFooler attack success rate from 98.7% to 92.3%.
- Efficiency: The Geometry Attack is highly optimized, running at least 4 times faster than TextFooler and up to 10 times faster than PWWS, making on-the-fly adversarial generation feasible during pretraining.
Discussion and Insights
The success of ADCL demonstrates that labels are not strictly necessary for building model robustness. By grouping clean and adversarial examples based on semantic similarity, the model learns a more stable embedding space. Vector space analysis confirms that contrastive pretraining physically pulls clean and adversarial embeddings closer together, making it harder for attackers to push a representation across a decision boundary.
Conclusion and Future Work
ADCL provides a scalable path toward robust foundational models. While current experiments were limited by computational infrastructure to datasets like DBpedia, the authors plan to scale this method to massive web corpora like Wikipedia. This would determine if robustness gains scale alongside general performance, potentially leading to inherently more secure language models out of the box.
Summary updated