Adversarial Robustness via Adversarial Label-Smoothing

Morgane Goibert, Elvis DohmatobarXiv 2019 · OpenReview

TL;DR

This paper introduces Label-Smoothing (LS) as a novel and efficient method to improve the robustness of deep neural networks against adversarial attacks. By modifying the training objective, LS encourages models to be less overconfident, leading to better generalization and significantly improved adversarial accuracy without increasing training time or requiring architectural changes.

What Problem Does This Paper Address?

Deep neural networks (NNs), despite their impressive performance, are notoriously vulnerable to adversarial examples. These are subtly modified inputs that are imperceptible to humans but cause NNs to misclassify them. This vulnerability is a major concern for critical applications like autonomous driving and medical diagnosis. While many defense mechanisms exist, they often come with significant computational costs, architectural modifications, or a trade-off with standard accuracy. This paper aims to address this by proposing Label-Smoothing as an accessible and effective defense strategy.

Methods

The core contribution is a unified framework for Label-Smoothing (LS). LS works by softening the "hard" one-hot encoded labels used in standard training. Instead of assigning a probability of 1 to the correct class and 0 to all others, LS assigns a probability distribution over the classes that is "close" to the true label. The paper proposes several LS variants, including:

  • Adversarial Label-Smoothing (ALS): This method specifically targets the "worst-case" smooth label for each example, aiming to make the model robust against the most challenging perturbations.
  • Boltzmann Label-Smoothing (BLS): This variant interpolates between ALS and standard LS, offering a tunable parameter (temperature) to control the smoothing effect.
  • Second-Best Label-Smoothing (SBLS): This approach focuses on the "closest" wrong class, aiming to improve robustness by pushing the decision boundary away from dense regions.

These methods are implemented by modifying the cross-entropy loss function, making them easy to integrate into existing training pipelines. The paper also provides theoretical analysis showing that LS leads to "logit-squeezing," which inherently improves robustness.

Datasets

The authors evaluated their methods on several standard image classification datasets:

  • MNIST
  • CIFAR10
  • SVHN

They used various neural network architectures, including linear models, MLPs, LeNet, and ResNet.

Key Insights and Ideas

  • LS as a Geometry-Aware Regularizer: LS effectively encourages models to consider the underlying geometry of the data distribution, leading to decision boundaries that are more robust to small perturbations.
  • Logit-Squeezing: A key theoretical insight is that LS induces "logit-squeezing," meaning the differences between the logit values (pre-softmax outputs) for different classes are reduced. This makes the model less confident in its predictions and thus less susceptible to adversarial manipulation.
  • Efficiency and Simplicity: A major advantage of LS is its ease of implementation. It's a simple modification to the loss function, requiring no changes to the network architecture and incurring no significant increase in training time compared to standard training.
  • Balancing Accuracy and Robustness: The paper demonstrates that LS can improve adversarial accuracy without substantially sacrificing standard accuracy, addressing a common trade-off in adversarial defense.

Key Results

  • Improved Adversarial Robustness: Across various datasets and models, LS methods consistently improved adversarial accuracy against a range of attacks (FGSM, BIM, DeepFool, Carlini-Wagner). For instance, in experiments with MNIST LeNet, ALS and BLS showed significant gains in robustness.
  • Competitive with Adversarial Training: While adversarial training (a strong baseline) often performed slightly better on specific attacks like FGSM, LS methods were competitive and sometimes superior, especially against attacks like C&W.
  • Minimal Impact on Standard Accuracy: The paper shows that LS generally maintains or slightly improves standard accuracy, indicating it doesn't hinder the model's ability to learn from clean data.
  • ALS and BLS as Preferred Methods: The authors suggest that ALS and BLS generally yield better results than SLS and SBLS and can be considered default choices for implementing LS.

Limitations and Future Work

  • No Perfect Robustness: The authors acknowledge that LS does not provide perfect robustness but rather improves it significantly.
  • Further Exploration: They suggest that future work could involve exploring LS with a wider variety of NNs and datasets.
  • Deeper Understanding: While LS is shown to be effective, a deeper theoretical understanding of the adversarial phenomenon and learned representations in NNs would be beneficial.
  • Hyperparameter Tuning: While the temperature parameter for BLS didn't show a significant impact in their experiments, further investigation into optimal hyperparameter choices for different LS variants and tasks could be valuable.

Summary updated

Abstract

We study Label-Smoothing as a means for improving adversarial robustness of supervised deep-learning models. After establishing a thorough and unified framework, we propose several variations to this general method: adversarial, Boltzmann and second-best Label-Smoothing methods, and we explain how to construct your own one. On various datasets (MNIST, CIFAR10, SVHN) and models (linear models, MLPs, LeNet, ResNet), we show that Label-Smoothing in general improves adversarial robustness against a variety of attacks (FGSM, BIM, DeepFool, Carlini-Wagner) by better taking account of the dataset geometry. The proposed Label-Smoothing methods have two main advantages: they can be implemented as a modified cross-entropy loss, thus do not require any modifications of the network architecture nor do they lead to increased training times, and they improve both standard and adversarial accuracy.
BibTeX
@misc{goibert2019adversarial,
  author = {Morgane Goibert and Elvis Dohmatob},
  title = {{Adversarial Robustness via Adversarial Label-Smoothing}},
  year = {2019},
  eprint = {1906.11567},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.11567},
}
Figure