Introduction
Fine-tuning large-scale pre-trained language models (LLMs) like BERT and RoBERTa has become the standard approach for downstream NLP tasks. However, these models often suffer from overfitting, which degrades their generalization on unseen data. Adversarial Training (AT)—training on perturbed data—is a common defense, but traditional methods treat the interaction between the model and the adversary as a deterministic, pure-strategy game. A recent study presented at IJCAI-23 introduces MAT (Mixed-strategy Adversarial Training), proposing that a probabilistic approach is key to unlocking better robustness and generalization.
The Mixed-Strategy Framework
In standard AT, the model and adversary engage in a "pure-strategy" game where each chooses a single, deterministic move (parameter updates and perturbations). The authors argue this is limiting because a stable Nash equilibrium may not exist in such a constrained space, leading to unstable training.
MAT re-imagines this interaction as a mixed-strategy game. Instead of selecting fixed values, both players operate over probability distributions. This is analogous to playing Rock-Paper-Scissors: a pure strategy (always playing "Rock") is easily defeated, whereas a mixed strategy (randomizing between Rock, Paper, and Scissors) makes a player unpredictable and harder to beat.
Methodology: From Theory to Practice
To solve this probabilistic game, MAT employs several key techniques:
- Entropy Mirror Descent (EMD): This optimization method uses Shannon entropy as a distance measure, naturally accounting for the uncertainty in mixed strategies and providing theoretical guarantees for reaching a Nash equilibrium.
- Stochastic Gradient Langevin Dynamics (SGLD): Since computing full distributions for LLMs is intractable, SGLD is used to efficiently sample from the perturbation and parameter distributions.
- Exponential Moving Averages (EMA): EMA tracks the "mean" strategy of the model during training, stabilizing the learning process.
Experimental Evaluation
The researchers evaluated MAT on two primary benchmarks:
- GLUE (General Language Understanding Evaluation): Nine tasks measuring general linguistic competence.
- ANLI (Adversarial Natural Language Inference): A challenging benchmark designed to fool models iteratively.
Key Results
MAT consistently outperformed state-of-the-art adversarial methods like FreeLB and SMART. On BERT-base, MAT achieved an overall GLUE score of 85.11, and on RoBERTa-large, it reached 90.35. The model showed exceptional robustness on ANLI Round 3, achieving 49.3% accuracy—nearly 2 points higher than the SMART baseline.
Beyond raw scores, MAT demonstrated high reliability. Error bar analysis indicates that even the worst-performing runs of MAT often exceeded the best runs of competing methods across different random seeds, highlighting its stability.
Conclusion and Future Work
By expanding the strategy space of adversarial training, MAT provides a more robust framework for fine-tuning LLMs. While it introduces some computational overhead due to the sampling process, the gains in generalization and stability are significant. Future research may focus on optimizing the training loop efficiency and extending this mixed-strategy approach to other domains like computer vision.
Summary updated