Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach

Simiao Zuo, Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Jianfeng Gao, Weizhu Chen, Tuo Zhao

Abstract

Adversarial Regularization (AR) has emerged as a powerful technique for improving the generalization of deep learning models. However, standard AR formulations typically rely on a zero-sum min-max game, which treats the model and the attacker as equal participants. This symmetry often results in "over-strong" attacks that degrade the model's performance on clean data. We propose Stackelberg Adversarial Regularization (SALT), which reframes the AR process as a Stackelberg game. In this framework, the model acts as a leader that anticipates the attacker's moves. By employing unrolled optimization and Hessian-vector products, SALT achieves superior generalization and model calibration. Experimental results across Natural Language Understanding (NLU) and Neural Machine Translation (NMT) tasks demonstrate that SALT consistently outperforms state-of-the-art AR baselines.

Introduction

The goal of Adversarial Regularization is to find model parameters that are robust to small input perturbations. Traditionally, this is modeled as a min-max problem where an attacker maximizes loss and a defender minimizes it. While effective, this zero-sum setup assumes the attacker and defender have equal priority. In practice, this can lead to over-regularization, where the model focuses so much on defending against aggressive perturbations that it fails to fit the original, clean data distribution. This phenomenon is particularly acute in Natural Language Processing (NLP), where the discrete nature of text makes defining appropriate perturbation sets challenging.

The SALT Framework: A Strategic Approach

SALT shifts the paradigm from a simultaneous zero-sum game to a hierarchical Stackelberg game. In this formulation, the model (the leader) optimizes its parameters by anticipating the optimal response of the perturbation generator (the follower).

Unrolled Optimization

To implement this anticipation, SALT utilizes unrolled optimization. The leader's gradient calculation involves differentiating through the follower's optimization steps. This "look-ahead" mechanism allows the model to find a more stable optimization path. To manage the computational complexity of second-order derivatives, the authors employ Hessian-vector products (Hvp) via finite differences, ensuring that the training overhead remains manageable—comparable to approximately two standard backpropagation passes.

SALT demonstrates superior stability and performance across various hyperparameters, including unrolling steps (K) and perturbation strengths (epsilon).

Experimental Results

The researchers evaluated SALT on a diverse set of NLP benchmarks, including the GLUE suite for NLU and several IWSLT/WMT datasets for NMT.

Performance on NLU and NMT

A BERT-base model trained with SALT achieved an average score of 84.5 on the GLUE development set, surpassing the performance of a vanilla BERT-large model (84.0) despite having significantly fewer parameters. In translation tasks, SALT provided improvements of up to 2.0 BLEU points over standard Transformer baselines in low-resource settings.

Reducing Optimization Bias

One of the primary advantages of SALT is its ability to fit clean data more effectively than standard AR methods like SMART. By anticipating the attack rather than just reacting to it, the model avoids the underfitting typically caused by over-strong adversarial perturbations.

Comparison of training and validation loss curves between SALT and SMART. SALT achieves lower loss on both clean training and validation data, indicating reduced optimization bias.

Robustness and Calibration

SALT exhibits remarkable stability with respect to hyperparameters. As shown in the sensitivity analysis, SALT maintains high performance even as the perturbation strength increases, whereas competing methods often see performance degradation. Furthermore, SALT-trained models are better calibrated, meaning their predicted confidence levels more accurately reflect their actual accuracy, reducing the common issue of overconfidence in deep neural networks.

Conclusion

By reframing adversarial training as a Stackelberg game, SALT provides a more principled and effective way to regularize NLP models. The strategic advantage of anticipating the attacker allows for better fit on clean data, increased robustness to hyperparameters, and improved model calibration. While it introduces some computational overhead, the significant performance gains—such as BERT-base outperforming BERT-large—make it a compelling approach for large-scale NLP training.

Summary updated

Abstract

Adversarial regularization has been shown to improve the generalization performance of deep learning models in various natural language processing tasks. Existing works usually formulate the method as a zero-sum game, which is solved by alternating gradient descent/ascent algorithms. Such a formulation treats the adversarial and the defending players equally, which is undesirable because only the defending player contributes to the generalization performance. To address this issue, we propose Stackelberg Adversarial Regularization (SALT), which formulates adversarial regularization as a Stackelberg game. This formulation induces a competition between a leader and a follower, where the follower generates perturbations, and the leader trains the model subject to the perturbations. Different from conventional approaches, in SALT, the leader is in an advantageous position. When the leader moves, it recognizes the strategy of the follower and takes the anticipated follower's outcomes into consideration. Such a leader's advantage enables us to improve the model fitting to the unperturbed data. The leader's strategic information is captured by the Stackelberg gradient, which is obtained using an unrolling algorithm. Our experimental results on a set of machine translation and natural language understanding tasks show that SALT outperforms existing adversarial regularization baselines across all tasks. Our code is available at https://github.com/SimiaoZuo/Stackelberg-Adv.
BibTeX
@inproceedings{zuo2021adversarial,
  author = {Simiao Zuo and Chen Liang and Haoming Jiang and Xiaodong Liu and Pengcheng He and Jianfeng Gao and Weizhu Chen and Tuo Zhao},
  title = {{Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=noLRWzZerIX},
}
Figure