Abstract
Adversarial regularization has been shown to improve the generalization performance of deep learning models in various natural language processing tasks. Existing works usually formulate the method as a zero-sum game, which is solved by alternating gradient descent/ascent algorithms. Such a formulation treats the adversarial and the defending players equally, which is undesirable because only the defending player contributes to the generalization performance. To address this issue, we propose Stackelberg Adversarial Regularization (SALT), which formulates adversarial regularization as a Stackelberg game. This formulation induces a competition between a leader and a follower, where the follower generates perturbations, and the leader trains the model subject to the perturbations. Different from conventional approaches, in SALT, the leader is in an advantageous position. When the leader moves, it recognizes the strategy of the follower and takes the anticipated follower's outcomes into consideration. Such a leader's advantage enables us to improve the model fitting to the unperturbed data. The leader's strategic information is captured by the Stackelberg gradient, which is obtained using an unrolling algorithm. Our experimental results on a set of machine translation and natural language understanding tasks show that SALT outperforms existing adversarial regularization baselines across all tasks.
Introduction
Adversarial regularization (AR) is a powerful technique for improving the robustness and generalization of deep learning models. Traditionally, AR is framed as a min-max zero-sum game where an attacker generates perturbations to maximize loss, and a defender (the model) seeks to minimize it. However, this "fair fight" often lacks strategic foresight; the model typically reacts to the attacker's output without accounting for the attacker's underlying optimization process. This can lead to overly aggressive perturbations that hinder the model's ability to fit the original, clean data distribution.
Methods: The Stackelberg Game Formulation
To address these limitations, we propose SALT (Stackelberg Adversarial Regularization). SALT reframes adversarial training as a Stackelberg game, a leader-follower model where the model (leader) moves first by anticipating the follower's (attacker's) best response.
Key technical innovations include:
- Unrolled Optimization: The model "unrolls" the attacker's gradient ascent steps, allowing it to differentiate through the entire perturbation process.
- Stackelberg Gradient: This gradient captures the interaction between the leader and follower, enabling the model to account for how the attacker will adapt to its updates.
- Efficiency via Hvps: To handle the computational cost of second-order derivatives, we utilize Hessian-vector products (Hvp) via finite differences, making the approach scalable to large language models.
Experimental Evaluation
Optimization and Data Fitting
SALT demonstrates a clear advantage in fitting the underlying data distribution. By anticipating the attacker's strategy, the model avoids being overwhelmed by adversarial noise. As shown in the loss curves below, SALT achieves lower training and validation losses compared to the SMART baseline, indicating superior optimization.
Hyperparameter Stability
A core strength of SALT is its robustness to hyperparameter choices. While traditional methods like SMART can be highly sensitive to the perturbation strength ($\epsilon$) or the number of optimization steps ($K$), SALT maintains stable and high performance across a wide range of values.
Key Results across NLP Tasks
- Neural Machine Translation (NMT): SALT achieved up to a 2.0 BLEU score improvement over vanilla Transformers on low-resource datasets and consistently outperformed baselines like FreeLB and SMART.
- Natural Language Understanding (GLUE): A BERT-base model trained with SALT achieved an average score of 84.5, surpassing a vanilla BERT-large model despite having significantly fewer parameters. Notable gains were observed on small datasets like RTE (+9.4) and CoLA (+6.3).
- Calibration and Robustness: SALT models exhibit significantly lower Expected Calibration Error (ECE), providing more reliable confidence scores. It also outperformed SMART by 1.5% to 2% on the challenging ANLI dataset.
Conclusion
By treating adversarial training as a Stackelberg game, SALT provides the model with a strategic advantage that translates into better generalization, improved data fitting, and increased stability. Our results across NMT and NLU tasks demonstrate that SALT is a robust and effective framework for regularizing deep learning models in natural language processing.
Summary updated