Introduction
Adversarial Regularization (AR) has emerged as a critical technique for enhancing the generalization and robustness of Natural Language Processing (NLP) models. By introducing small, "worst-case" perturbations to input embeddings, AR encourages models to maintain stable outputs despite minor input variations. However, standard AR implementations face two significant hurdles: extreme computational overhead and optimization instability.
Conventional methods like Projected Gradient Descent (PGD) require multiple forward and backward passes per sample in every epoch, often slowing training by 3 to 10 times. Furthermore, recalculating perturbations every epoch against a shifting model leads to high gradient variance, which can hinder convergence. This paper introduces ARCH (Adversarial Regularization with CacHing), a framework designed to overcome these bottlenecks through efficient engineering and algorithmic stability.
The ARCH Framework
ARCH tackles the limitations of traditional AR through three core innovations:
- Perturbation Caching: Instead of re-calculating perturbations at every step, ARCH generates them once and stores them in a cache. These perturbations are reused across multiple training epochs and only updated periodically (e.g., every 15-20 epochs). This "stability over freshness" approach allows the model to adapt more deeply to specific noise patterns.
- Exponential Moving Average (EMA) Updates: When perturbations are eventually updated, ARCH employs an EMA strategy to blend new calculations with existing cached values. This smoothing mechanism prevents abrupt changes in the noise distribution, further stabilizing the training process.
- KNN-based Memory Optimization: To mitigate the massive RAM requirements of caching perturbations for large-scale datasets (which can exceed 100GB), ARCH utilizes a K-Nearest Neighbors (KNN) strategy. By caching only a small fraction (e.g., 1-10%) of perturbations and reconstructing others on-the-fly via neighbor averaging in the embedding space, ARCH maintains performance while drastically reducing the memory footprint.
Experimental Results
Computational Efficiency
ARCH achieves dramatic reductions in training time compared to state-of-the-art AR methods like SMART and FreeLB. On large-scale benchmarks such as WMT’16, ARCH reduced GPU hours from 576 (SMART) to just 176, representing a nearly 70% speedup.
Model Performance and Stability
Despite the use of "stale" cached perturbations, ARCH consistently outperforms baselines. In Neural Machine Translation (NMT) tasks, it provided an average improvement of roughly 0.5 BLEU points. On the GLUE benchmark, ARCH raised the average score by 0.7 over existing methods.
A key insight of this work is that caching reduces stochastic gradient variance. By exposing the model to consistent perturbations over several epochs, the optimization landscape becomes smoother and less volatile than methods that introduce fresh noise every epoch.
Conclusion
ARCH demonstrates that adversarial training does not have to be prohibitively expensive or unstable. By leveraging perturbation caching and KNN-based reconstruction, ARCH provides a scalable path for applying robust regularization to large-scale NLP models. The findings highlight that data-dependent, stable noise is often more informative for model generalization than high-variance, frequently updated perturbations.
Summary updated