Abstract
Adversarial training defense (ATD) and virtual adversarial training (VAT) are the two most effective methods to improve model robustness against attacks and model generalization. While ATD is usually applied in robust machine learning, VAT is used in semi-supervised learning and domain adaption. In this paper, we introduce a novel adversarial local distribution regularization. The adversarial local distribution is defined by a set of all adversarial examples within a ball constraint given a natural input. We illustrate this regularization is a general form of previous methods (e.g., PGD, TRADES, VAT and VADA). We conduct comprehensive experiments on MNIST, SVHN and CIFAR10 to illustrate that our method outperforms well-known methods such as PGD, TRADES and ADT in robust machine learning, VAT in semi-supervised learning and VADA in domain adaption.
Introduction
Deep neural networks are notoriously fragile. They are susceptible to adversarial attacks—tiny, purposeful perturbations that flip a model's prediction—and often struggle to generalize when training data is scarce (semi-supervised learning) or when the data distribution shifts (domain adaptation).
Existing defenses like Adversarial Training Defense (ATD) or Virtual Adversarial Training (VAT) usually focus on finding a single worst-case adversarial example to train against. However, this approach has several limitations:
- Limited Exploration: A single attack point doesn't capture the complexity of the local loss landscape.
- Lack of Diversity: Even multiple attacks with random restarts often "collapse" toward the same local maxima.
- Restrictive Assumptions: Previous attempts to model a distribution of perturbations (like ADT) assumed the distribution followed a simple Gaussian shape, which rarely holds true in complex high-dimensional spaces.
Methodology: Adversarial Local Distribution (ALD)
To address these issues, we propose Adversarial Local Distribution (ALD) regularization. Instead of searching for one point, we define a distribution of all possible adversarial examples within a small "ball" around a natural input.
The key technical innovation is the use of Stein Variational Gradient Descent (SVGD) to approximate this distribution. The method maintains a set of $n$ "particles" (potential adversarial examples). The SVGD update rule contains two terms: one that pushes particles toward high-loss areas (making them "adversarial") and a repulsive term that pushes particles away from each other.
By minimizing the ALD loss, the model effectively maximizes the entropy of the adversarial distribution. This forces the model to produce smooth, consistent outputs across the entire local neighborhood, rather than just at a few specific points. ALD is a "general form" of several famous algorithms; by changing the loss function or reducing the particle count to one, ALD mathematically reduces to PGD, TRADES, VAT, or VADA.
Experimental Results
We evaluated ALD on three standard computer vision benchmarks: MNIST, SVHN, and CIFAR10 across three distinct settings: robust machine learning, semi-supervised learning, and domain adaptation.
Robustness and Generalization
On CIFAR10, "Our_TRADES" (the ALD version) achieved a robust accuracy of 53.9% against PGD-200 attacks, significantly higher than the standard TRADES (52.5%) and the Gaussian-based ADT (46.1%). The "aha moment" is that a model trained against a diverse set of perturbations is much harder to fool than one trained against a single "strongest" perturbation.
Semi-Supervised Learning
In the CIFAR10-1000 setting (using only 1000 labeled images), ALD improved accuracy by 6% over the baseline and consistently outperformed standard VAT as the number of particles increased.
Domain Adaptation
When transferring from SVHN to MNIST, ALD consistently maintained higher accuracy than the state-of-the-art VADA method. While more complex than single-point methods, the running time of ALD scales linearly with the number of particles, making it computationally feasible on modern GPUs.
Discussion and Future Work
While ALD provides significant gains, it introduces new hyperparameters, such as the kernel width and the number of particles ($n$), which require empirical tuning. We also observed that using too many particles (e.g., $n=24$ for MNIST) can lead to over-regularization, potentially hurting natural accuracy. Future research is needed to see how this particle-based approach scales to massive architectures like Transformers or higher-resolution datasets like ImageNet.
Our implementation is available on GitHub: https://github.com/PotatoThanh/ALD-Regularization
Summary updated