Achieving Generalizable Robustness of Deep Neural Networks by Stability Training

Jan Laermann, Wojciech Samek, Nils Strodthoff

Abstract

We study the recently introduced stability training as a general-purpose method to increase the robustness of deep neural networks against input perturbations. In particular, we explore its use as an alternative to data augmentation and validate its performance against a number of distortion types and transformations including adversarial examples. In our image classification experiments using ImageNet data, stability training performs on a par or even outperforms data augmentation for specific transformations, while consistently offering improved robustness against a broader range of distortion strengths and types unseen during training, a considerably smaller hyperparameter dependence, and fewer potentially negative side effects compared to data augmentation.

Introduction

Deep Neural Networks (DNNs) are notoriously brittle. While they can outperform humans on clean datasets, their performance often collapses when faced with small "distortions"—anything from Gaussian noise and JPEG compression artifacts to simple rotations or adversarial attacks.

The standard industry fix is Data Augmentation (DA): feeding the network noisy versions of images during training so it "learns" the noise. However, DA suffers from three major flaws:

  1. Specificity: Robustness gained via DA is often highly specific to the exact type and strength of noise used during training.
  2. Performance Trade-offs: Training heavily on distorted data can degrade a model's accuracy on clean, unperturbed images.
  3. Hyperparameter Sensitivity: Finding the right balance of augmentation frequency and intensity is a tedious, manual process.

Methodology: Stability Training (ST)

The core idea of Stability Training is to decouple the task of classification from the task of robustness. Instead of training on noisy data directly, ST enforces a consistency constraint between clean and perturbed inputs. The network is trained using a composite loss function:

$$L(x, x', \hat{y}; \theta) = L_0(x, \hat{y}; \theta) + \alpha L_{stab}(x, x'; \theta)$$

  • $L_0$: The standard classification loss calculated only on the clean image ($x$).
  • $L_{stab}$: A consistency constraint (typically Kullback-Leibler divergence) that forces the network to produce similar probability distributions for both the clean image ($x$) and its distorted version ($x'$).

For transformations like rotations, where a rotated image isn't necessarily "noisier" than the original, the authors introduce a Symmetrical Stability Objective using symmetrized KL divergence to ensure the network treats both versions with equal importance.

Distortion types and their practical parameter values used in the study.

Experimental Setup: The TIFS Dataset

To evaluate these methods, the researchers introduced the Tiny ImageNet Full-Sized (TIFS) benchmark. Derived from ILSVRC2012, TIFS features 200 classes with 450 training samples each. Unlike standard Tiny ImageNet, which downsamples images to 64x64, TIFS preserves the full 224x224 resolution. This allows for realistic testing of resolution-dependent distortions like JPEG compression and thumbnail resizing while maintaining computational efficiency.

Key Results and Discussion

Stability training demonstrates superior generalization and lower risk compared to traditional data augmentation across several scenarios.

Generalization and Cross-Distortion Performance

In "Gauss $\rightarrow$ Gauss" tests, ST outperformed DA across almost the entire spectrum of noise intensity. More importantly, ST avoids the "catastrophic forgetting" associated with DA. When trained on thumbnail resizing and tested on JPEG compression, DA models suffered a ~10% drop in accuracy compared to the baseline. ST models, however, maintained baseline performance, showing no negative side effects from training on an unrelated distortion.

Comparison of Stability Training (ST) and Data Augmentation (DA) on Gaussian noise (a) and cross-distortion scenarios (b).

Adversarial and Geometric Robustness

ST showed significant improvements against Fast Gradient Sign Method (FGSM) adversarial attacks compared to baseline and DA models, even without explicit adversarial training. Furthermore, the Symmetrical ST objective significantly outperformed both standard ST and DA for image rotations, particularly in the practical range of 0 to 30 degrees.

Conclusion

Stability training offers a more reliable and generalizable path to robust DNNs. By asking the model to find a "smooth" underlying representation invariant to change rather than memorizing specific noisy patterns, ST achieves better generalization across distortion strengths and types. While it doubles the memory requirements and increases training time per epoch, the reduced hyperparameter sensitivity and improved cross-distortion performance make it a compelling alternative to standard data augmentation.

Summary updated

Abstract

We study the recently introduced stability training as a general-purpose method to increase the robustness of deep neural networks against input perturbations. In particular, we explore its use as an alternative to data augmentation and validate its performance against a number of distortion types and transformations including adversarial examples. In our image classification experiments using ImageNet data stability training performs on a par or even outperforms data augmentation for specific transformations, while consistently offering improved robustness against a broader range of distortion strengths and types unseen during training, a considerably smaller hyperparameter dependence and less potentially negative side effects compared to data augmentation.
BibTeX
@inproceedings{laermann2019achieving,
  author = {Jan Laermann and Wojciech Samek and Nils Strodthoff},
  title = {{Achieving Generalizable Robustness of Deep Neural Networks by Stability Training}},
  booktitle = {GCPR},
  year = {2019},
  url = {https://openreview.net/forum?id=A1CU8ZK9AyT},
}
Figure