Impact of Adversarial Training on Robustness and Generalizability of Language Models

Enes Altinisik, Hassan Sajjad, Husrev Taha Sencar, Safa Messaoud, Sanjay Chawla

Introduction

Language Models (LMs) have demonstrated remarkable capabilities across natural language tasks but remain notoriously fragile. Minor, often imperceptible changes to input text—such as replacing a word with a synonym—can cause model accuracy to drop by as much as 90%. While Adversarial Training (AT) is a common defense mechanism, it often introduces a "robustness-generalization trade-off" in domains like Computer Vision, where making a model tougher typically degrades its performance on clean data. This paper investigates this trade-off within the context of NLP, comparing different AT strategies to determine which best balances model toughness with real-world intelligence.

Methodology

The researchers categorized adversarial training into three distinct families based on where the perturbations are applied:

  1. AT-DA (Data Augmentation): Static generation of adversarial examples (e.g., via back-translation) prior to the training process.
  2. AT-IP (Input-space Perturbation): Dynamic generation of attacks during training by searching for discrete word substitutions that maximize model loss.
  3. AT-EP (Embedding-space Perturbation): Adding continuous noise directly to word vectors (embeddings). This includes PGD-AT, which minimizes worst-case loss, and Local Distribution Smoothness (LDS), a regularization method.
The experimental framework illustrating the fine-tuning approaches and the subsequent evaluation of generalizability and robustness.

The evaluation utilized BERT-base models trained on sentiment classification tasks (IMDB and Movie Reviews) and tested on out-of-distribution (OOD) data from YELP to measure generalizability.

Key Insights: Robustness vs. Generalization

The study reveals a significant divergence in the utility of AT methods. If the primary goal is to prevent "hacking" by word-swapping tools like TextFooler, Input-space (AT-IP) methods are superior, effectively lowering the Attack Success Rate (ASR). For instance, on the Movie Reviews dataset, AT-IP reduced the average ASR from 62.0% to 56.1%.

However, for general performance, the researchers found an "NLP Exception." Unlike other fields, embedding-space AT (specifically PGD-AT) actually improved accuracy on both clean and OOD data. PGD-AT achieved a peak accuracy of 92.11% on the YELP dataset, outperforming standard BERT (91.24%) and all input-space methods. Furthermore, AT-EP is significantly more efficient, running only 1.5x slower than standard training, compared to the 10x-20x slowdown associated with AT-IP.

Mechanistic Analysis: The Role of Specialized Neurons

To understand why embedding-space perturbations enhance generalization, the authors conducted a Linguistic Correlation Analysis (LCA). This process identifies specific neurons within the transformer architecture responsible for recognizing linguistic properties like Parts-of-Speech (POS) tags.

The Linguistic Correlation Analysis (LCA) pipeline used to identify and rank neurons based on their activation for specific linguistic tags.

The analysis revealed that models trained with AT-EP developed neurons that were significantly more "focused" on specific POS tags (e.g., neurons firing exclusively for comparative adjectives). This suggests that embedding-space perturbations force the model to move beyond surface-level pattern matching and instead learn robust underlying linguistic structures.

Conclusion

This research demonstrates that adversarial training in NLP does not necessarily require a sacrifice in generalizability. While input-space perturbations are necessary for maximum robustness against discrete attacks, embedding-space perturbations offer a "sweet spot"—improving out-of-distribution performance and fostering the development of specialized linguistic neurons with relatively low computational overhead.

Summary updated

BibTeX
@inproceedings{altinisik2023impact,
  author = {Enes Altinisik and Hassan Sajjad and Husrev Taha Sencar and Safa Messaoud and Sanjay Chawla},
  title = {{Impact of Adversarial Training on Robustness and Generalizability of Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.496},
}
Figure