Adversarial Training as Stackelberg Game: An Unrolled Optimization Approach

Simiao Zuo, Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Jianfeng Gao, Weizhu Chen, Tuo Zhao

Abstract

Adversarial regularization has been shown to improve the generalization performance of deep learning models in various natural language processing tasks. Existing works usually formulate the method as a zero-sum game, which is solved by alternating gradient descent/ascent algorithms. Such a formulation treats the adversarial and the defending players equally, which is undesirable because only the defending player contributes to the generalization performance. To address this issue, we propose Stackelberg Adversarial Regularization (SALT), which formulates adversarial regularization as a Stackelberg game. This formulation induces a competition between a leader and a follower, where the follower generates perturbations, and the leader trains the model subject to the perturbations. Different from conventional approaches, in SALT, the leader is in an advantageous position. When the leader moves, it recognizes the strategy of the follower and takes the anticipated follower's outcomes into consideration. Such a leader's advantage enables us to improve the model fitting to the unperturbed data. The leader's strategic information is captured by the Stackelberg gradient, which is obtained using an unrolling algorithm. Our experimental results on a set of machine translation and natural language understanding tasks show that SALT outperforms existing adversarial regularization baselines across all tasks.

Introduction

Adversarial regularization (AR) is a powerful technique for improving the robustness and generalization of deep learning models. Traditionally, AR is framed as a min-max zero-sum game where an attacker generates perturbations to maximize loss, and a defender (the model) seeks to minimize it. However, this "fair fight" often lacks strategic foresight; the model typically reacts to the attacker's output without accounting for the attacker's underlying optimization process. This can lead to overly aggressive perturbations that hinder the model's ability to fit the original, clean data distribution.

Methods: The Stackelberg Game Formulation

To address these limitations, we propose SALT (Stackelberg Adversarial Regularization). SALT reframes adversarial training as a Stackelberg game, a leader-follower model where the model (leader) moves first by anticipating the follower's (attacker's) best response.

Key technical innovations include:

  • Unrolled Optimization: The model "unrolls" the attacker's gradient ascent steps, allowing it to differentiate through the entire perturbation process.
  • Stackelberg Gradient: This gradient captures the interaction between the leader and follower, enabling the model to account for how the attacker will adapt to its updates.
  • Efficiency via Hvps: To handle the computational cost of second-order derivatives, we utilize Hessian-vector products (Hvp) via finite differences, making the approach scalable to large language models.

Experimental Evaluation

Optimization and Data Fitting

SALT demonstrates a clear advantage in fitting the underlying data distribution. By anticipating the attacker's strategy, the model avoids being overwhelmed by adversarial noise. As shown in the loss curves below, SALT achieves lower training and validation losses compared to the SMART baseline, indicating superior optimization.

Training and validation loss curves for SALT vs SMART.

Hyperparameter Stability

A core strength of SALT is its robustness to hyperparameter choices. While traditional methods like SMART can be highly sensitive to the perturbation strength ($\epsilon$) or the number of optimization steps ($K$), SALT maintains stable and high performance across a wide range of values.

Sensitivity analysis of SALT vs SMART across K and epsilon.

Key Results across NLP Tasks

  • Neural Machine Translation (NMT): SALT achieved up to a 2.0 BLEU score improvement over vanilla Transformers on low-resource datasets and consistently outperformed baselines like FreeLB and SMART.
  • Natural Language Understanding (GLUE): A BERT-base model trained with SALT achieved an average score of 84.5, surpassing a vanilla BERT-large model despite having significantly fewer parameters. Notable gains were observed on small datasets like RTE (+9.4) and CoLA (+6.3).
  • Calibration and Robustness: SALT models exhibit significantly lower Expected Calibration Error (ECE), providing more reliable confidence scores. It also outperformed SMART by 1.5% to 2% on the challenging ANLI dataset.

Conclusion

By treating adversarial training as a Stackelberg game, SALT provides the model with a strategic advantage that translates into better generalization, improved data fitting, and increased stability. Our results across NMT and NLU tasks demonstrate that SALT is a robust and effective framework for regularizing deep learning models in natural language processing.

Summary updated

Abstract

Adversarial regularization has been shown to improve the generalization performance of deep learning models in various natural language processing tasks. Existing works usually formulate the method as a zero-sum game, which is solved by alternating gradient descent/ascent algorithms. Such a formulation treats the adversarial and the defending players equally, which is undesirable because only the defending player contributes to the generalization performance. To address this issue, we propose Stackelberg Adversarial Regularization (SALT), which formulates adversarial regularization as a Stackelberg game. This formulation induces a competition between a leader and a follower, where the follower generates perturbations, and the leader trains the model subject to the perturbations. Different from conventional approaches, in SALT, the leader is in an advantageous position. When the leader moves, it recognizes the strategy of the follower and takes the anticipated follower's outcomes into consideration. Such a leader's advantage enables us to improve the model fitting to the unperturbed data. The leader's strategic information is captured by the Stackelberg gradient, which is obtained using an unrolling algorithm. Our experimental results on a set of machine translation and natural language understanding tasks show that SALT outperforms existing adversarial regularization baselines across all tasks. Our code is available at https://github.com/SimiaoZuo/Stackelberg-Adv.
BibTeX
@inproceedings{zuo2021adversarial,
  author = {Simiao Zuo and Chen Liang and Haoming Jiang and Xiaodong Liu and Pengcheng He and Jianfeng Gao and Weizhu Chen and Tuo Zhao},
  title = {{Adversarial Training as Stackelberg Game: An Unrolled Optimization Approach}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=noLRWzZerIX},
}
Figure