MMA Training: Direct Input Space Margin Maximization through Adversarial Training

Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, Ruitong HuangICLR 2020 · OpenReview

Deep neural networks are notoriously fragile. Tiny, often imperceptible "adversarial perturbations" can easily flip a model's prediction—for instance, tricking it into classifying a panda as a bucket. The standard defense against this fragility is Adversarial Training (AT), which involves training the model on examples that have been perturbed within a fixed-size neighborhood, defined by a radius $\epsilon$.

However, standard AT suffers from a fundamental flaw: the choice of $\epsilon$ is arbitrary and applied uniformly. If $\epsilon$ is too small, the model lacks meaningful robustness; if it's too large, the model's accuracy on clean, unperturbed data collapses. Furthermore, not all data points are equally "easy" to protect. Some images naturally sit closer to the classifier's decision boundary than others. Forcing a single, fixed $\epsilon$ across the entire dataset is mathematically suboptimal and often leads to a poor trade-off between accuracy and robustness.

Max-Margin Adversarial (MMA) Training

To address the "one-size-fits-all" limitation of standard AT, the authors propose Max-Margin Adversarial (MMA) Training. Instead of using a fixed $\epsilon$, MMA aims to directly maximize the margin—defined as the minimum distance an input $x$ must travel to cross the decision boundary.

Illustration of the margin and the shortest successful perturbation.

The authors mathematically demonstrate that maximizing this margin is equivalent to minimizing the classification loss at the "shortest successful perturbation" (denoted as $\delta^*$). To achieve this in practice, MMA introduces several key components:

  • AN-PGD (Adaptive Norm PGD): A novel attack algorithm that specifically searches for the point on the decision boundary closest to the original input, rather than just finding any adversarial example within a set radius.
  • Soft Logit Margin (SLM) Loss: A smooth "LogSumExp" surrogate that replaces the discrete "max" function in the logit margin, stabilizing the training process.
  • Hinge Threshold ($d_{max}$): A threshold ensuring the model doesn't waste capacity trying to push margins beyond a point that is already "robust enough," or trying to solve impossibly hard cases.
  • Clean Loss Integration: Pure margin maximization can make the loss landscape too flat, hindering the adversary's ability to find the boundary. Adding a standard "clean" cross-entropy loss term keeps the landscape tilted, ensuring gradient-based attacks can still operate effectively during training.
Comparison of loss function updates using the shortest successful perturbation versus a fixed epsilon.

Key Insights and Results

The method was rigorously evaluated on standard computer vision benchmarks (MNIST and CIFAR10) against both $L_\infty$ and $L_2$ norm-constrained perturbations. The results highlight several critical advantages of the MMA approach:

  • The Adaptive Advantage: By providing an "individualized" $\epsilon$ for every data point, MMA allows the model to push the boundary as far as possible for easy points without collapsing its performance on hard points.
  • Theoretical Grounding: The paper provides a rigorous theoretical link showing that standard adversarial training is actually just maximizing either a lower or upper bound of the margin, depending on whether the fixed $\epsilon$ is smaller or larger than the true margin.
  • Superior Average Robustness: On CIFAR10 ($L_\infty$), the MMA-32 model achieved a clean accuracy of 84.36% and a robust accuracy of 47.18% at $\epsilon=8/255$. Furthermore, a single MMA-trained model matched or exceeded the performance of computationally expensive ensembles of standard adversarially trained models.
  • Graceful Failure: Standard PGD training is highly sensitive to its hyperparameters. When pushed to a high $\epsilon$ (e.g., 24/255), standard AT accuracy dropped to a near-useless 10.9%. Under similar conditions, MMA maintained high clean accuracy and better robustness across all tested attack scales, proving highly stable with respect to its primary hyperparameter ($d_{max}$).

Limitations and Future Work

While MMA offers a more principled approach to adversarial robustness, it does come with trade-offs. Finding the "shortest successful perturbation" ($\delta^*$) requires bisection searches, making it more computationally demanding than the standard PGD attacks used in traditional AT. Additionally, finding the true $\delta^*$ is a non-convex problem; while AN-PGD works well in practice, it cannot guarantee finding the absolute global minimum distance to the boundary.

Looking forward, the authors note that MMA focuses primarily on the direction of the decision boundary. Combining MMA with calibration techniques could yield even stronger, more reliable defenses in the future.

Summary updated

Abstract

We study adversarial robustness of neural networks from a margin maximization perspective, where margins are defined as the distances from inputs to a classifier's decision boundary. Our study shows that maximizing margins can be achieved by minimizing the adversarial loss on the decision boundary at the "shortest successful perturbation", demonstrating a close connection between adversarial losses and the margins. We propose Max-Margin Adversarial (MMA) training to directly maximize the margins to achieve adversarial robustness. Instead of adversarial training with a fixed $\epsilon$, MMA offers an improvement by enabling adaptive selection of the "correct" $\epsilon$ as the margin individually for each datapoint. In addition, we rigorously analyze adversarial training with the perspective of margin maximization, and provide an alternative interpretation for adversarial training, maximizing either a lower or an upper bound of the margins. Our experiments empirically confirm our theory and demonstrate MMA training's efficacy on the MNIST and CIFAR10 datasets w.r.t. $\ell_\infty$ and $\ell_2$ robustness.
BibTeX
@inproceedings{ding2020mma,
  author = {Gavin Weiguang Ding and Yash Sharma and Kry Yik Chau Lui and Ruitong Huang},
  title = {{MMA Training: Direct Input Space Margin Maximization through Adversarial Training}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=7EtZu6dxiNM},
}
Figure