MAT: Mixed-Strategy Game of Adversarial Training in Fine-tuning

Zhehua Zhong, Tianyi Chen, Zhen Wang

Introduction

Fine-tuning large-scale pre-trained language models (LLMs) like BERT and RoBERTa has become the standard approach for downstream NLP tasks. However, these models often suffer from overfitting, which degrades their generalization on unseen data. Adversarial Training (AT)—training on perturbed data—is a common defense, but traditional methods treat the interaction between the model and the adversary as a deterministic, pure-strategy game. A recent study presented at IJCAI-23 introduces MAT (Mixed-strategy Adversarial Training), proposing that a probabilistic approach is key to unlocking better robustness and generalization.

The Mixed-Strategy Framework

In standard AT, the model and adversary engage in a "pure-strategy" game where each chooses a single, deterministic move (parameter updates and perturbations). The authors argue this is limiting because a stable Nash equilibrium may not exist in such a constrained space, leading to unstable training.

MAT re-imagines this interaction as a mixed-strategy game. Instead of selecting fixed values, both players operate over probability distributions. This is analogous to playing Rock-Paper-Scissors: a pure strategy (always playing "Rock") is easily defeated, whereas a mixed strategy (randomizing between Rock, Paper, and Scissors) makes a player unpredictable and harder to beat.

Comparison between pure-strategy and mixed-strategy games using the Rock-Paper-Scissors analogy.

Methodology: From Theory to Practice

To solve this probabilistic game, MAT employs several key techniques:

  1. Entropy Mirror Descent (EMD): This optimization method uses Shannon entropy as a distance measure, naturally accounting for the uncertainty in mixed strategies and providing theoretical guarantees for reaching a Nash equilibrium.
  2. Stochastic Gradient Langevin Dynamics (SGLD): Since computing full distributions for LLMs is intractable, SGLD is used to efficiently sample from the perturbation and parameter distributions.
  3. Exponential Moving Averages (EMA): EMA tracks the "mean" strategy of the model during training, stabilizing the learning process.

Experimental Evaluation

The researchers evaluated MAT on two primary benchmarks:

  • GLUE (General Language Understanding Evaluation): Nine tasks measuring general linguistic competence.
  • ANLI (Adversarial Natural Language Inference): A challenging benchmark designed to fool models iteratively.

Key Results

MAT consistently outperformed state-of-the-art adversarial methods like FreeLB and SMART. On BERT-base, MAT achieved an overall GLUE score of 85.11, and on RoBERTa-large, it reached 90.35. The model showed exceptional robustness on ANLI Round 3, achieving 49.3% accuracy—nearly 2 points higher than the SMART baseline.

Performance consistency of MAT across various GLUE benchmark tasks.

Beyond raw scores, MAT demonstrated high reliability. Error bar analysis indicates that even the worst-performing runs of MAT often exceeded the best runs of competing methods across different random seeds, highlighting its stability.

Conclusion and Future Work

By expanding the strategy space of adversarial training, MAT provides a more robust framework for fine-tuning LLMs. While it introduces some computational overhead due to the sampling process, the gains in generalization and stability are significant. Future research may focus on optimizing the training loop efficiency and extending this mixed-strategy approach to other domains like computer vision.

Summary updated

Abstract

Fine-tuning large-scale models from pre-trained checkpoints has been demonstrated effective for various natural language processing (NLP) tasks. Previous works reveal that leveraging adversarial training methods during the fine-tuning stage significantly enhances the generalization and robustness of the models. However, from the perspective of optimization, the previous adversarial training methods suffer from converging onto local optima due to the non-convexity of the objective. In this work, we reformulate the adversarial training in the view of mixed strategy in game theory and incorporate full strategy space to avoid trapping in local stationarity. Methodologically, we derive the Nash equilibrium of mixed-strategy for adversarial training using entropy mirror descent to establish a novel mixed-strategy adversarial training algorithm (MAT). Numerically, to verify the effectiveness of MAT, we conducted extensive benchmark experiments over the large-scale pre-trained models such as BERT and RoBERTa. The experimental results show that MAT outperforms the previous state-of-the-art on both GLUE and ANLI benchmarks in terms of generalization and robustness.
BibTeX
@inproceedings{zhong2023mat,
  author = {Zhehua Zhong and Tianyi Chen and Zhen Wang},
  title = {{MAT: Mixed-Strategy Game of Adversarial Training in Fine-tuning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/520},
}
Figure