Diverse Gaussian Noise Consistency Regularization for Robustness and Uncertainty Calibration

Theodoros Tsiligkaridis, Athanasios TsiligkaridisIJCNN 2023 · OpenReview

TL;DR

This paper introduces DiGN, a novel regularization technique that improves the robustness and uncertainty calibration of deep neural networks against various image corruptions, particularly digital noise. By enforcing consistency between clean and noisy image representations, DiGN leads to more stable and reliable models, outperforming existing methods on several benchmarks.

What Problem Does This Paper Address?

Deep neural networks (DNNs) excel when training and testing data distributions align. However, in real-world applications, images are often subject to various corruptions (e.g., noise, blur, weather changes) that can drastically degrade model performance. This "domain shift" is a significant challenge. While adversarial training offers some robustness, it often focuses on imperceptible pixel-level perturbations and can hurt clean accuracy. Existing data augmentation techniques also struggle with certain types of noise and can be computationally expensive. The paper specifically targets the problem of digital noise corruptions, which are common due to hardware imperfections, and aims to improve both the accuracy of predictions and the reliability of uncertainty estimates in the presence of these unforeseen shifts.

Methods

The core contribution is Diverse Gaussian Noise Consistency Regularization (DiGN). The method works by:

  • Diverse Gaussian Noise Augmentation: During training, for each input image, multiple augmented versions are created by adding Gaussian noise. Crucially, the scale of this noise is sampled uniformly from a range [0, σmax], introducing diversity.
  • Consistency Regularization: A Kullback-Leibler (KL) divergence loss is used to enforce that the output probability distributions of the clean image and its noisy counterparts are similar. This encourages the model to produce consistent predictions despite the added noise.
  • Local Loss Landscape Analysis: The authors provide theoretical analysis showing that DiGN's consistency regularization has a dual effect on the model's loss landscape: it smooths the landscape by minimizing curvature and flattens it by minimizing the gradient norm. This is shown to be beneficial for robustness and uncertainty calibration.

DiGN is presented as a plug-and-play regularization technique that can be added to existing training pipelines with minimal modifications.

Datasets

  • Training and Evaluation: CIFAR-10, CIFAR-100, and Tiny-ImageNet.
  • Corrupted Benchmarks: CIFAR-10-C, CIFAR-100-C, and Tiny-ImageNet-C. These datasets are standard benchmarks for evaluating robustness against common corruptions, including weather, digital, blur, and noise. The "gaussian noise" corruption was excluded from some evaluations for fairness.

Key Insights and Ideas

  • Diverse Noise Scales Matter: Simply adding Gaussian noise isn't enough; varying the intensity of the noise during training is key to achieving broad robustness. This prevents the model from overfitting to a specific noise level.
  • Loss Landscape Geometry is Crucial: The theoretical analysis revealing DiGN's effect on the loss landscape (smoothing and flattening) provides a strong intuition for why it works. A smoother, flatter landscape makes the model less sensitive to small input perturbations.
  • Complementary to Data Augmentation: DiGN isn't just a standalone solution; it can be effectively combined with other diverse data augmentation techniques (like AugMix) to achieve even better performance, suggesting it addresses different aspects of robustness.
  • Bridging the Gap: DiGN helps close the performance gap between clean and corrupted data accuracy, a persistent challenge in robust machine learning.

Key Results

  • Improved Robustness Against Noise: DiGN significantly outperforms adversarial training and other strong data augmentation baselines on noise corruptions, achieving improvements of 4.2-18.4% on CIFAR-10, CIFAR-100, and Tiny-ImageNet.
  • Enhanced Uncertainty Calibration: When combined with state-of-the-art diverse data augmentations, DiGN improves robustness and uncertainty calibration by 3.7% and 5.5%, respectively, across all common corruptions (weather, digital, blur, noise).
  • State-of-the-Art Performance: The hybrid approach (DiGN + AugMix) achieves near state-of-the-art results on common corruptions, outperforming AugMix by significant margins in mean corrupted accuracy (mCA-N).
  • Model Resilience: Empirical validation shows DiGN models are more resilient to increasing perturbation strengths compared to standard training, maintaining higher validation accuracy and exhibiting flatter gradients and lower approximate curvature.

Limitations and Future Work

  • Computational Cost: While presented as plug-and-play, the generation of multiple augmented samples per input adds to the training time. The paper mentions that a "couple samples suffice to obtain strong robustness," implying a trade-off between the number of samples and performance.
  • Hyperparameter Tuning: The regularization parameter λ and the maximum noise scale σmax are hyperparameters that need to be tuned for optimal performance.
  • Future Work: The paper suggests that combining DiGN with other data augmentation strategies is a promising direction, which they demonstrate with the DiGN + AugMix hybrid. Further exploration of different noise distributions and their impact on the loss landscape could also be a future avenue.

Summary updated

Abstract

Deep neural networks achieve high prediction accuracy when the train and test distributions coincide. In practice though, various types of corruptions occur which deviate from this setup and cause severe performance degradations. Few methods have been proposed to address generalization in the presence of unforeseen domain shifts. In particular, digital noise corruptions arise commonly in practice during the image acquisition stage and present a significant challenge for current methods. In this paper, we propose a diverse Gaussian noise consistency regularization method for improving robustness of image classifiers under a variety of corruptions while still maintaining high clean accuracy. We derive bounds to motivate and understand the behavior of our Gaussian noise consistency regularization using a local loss landscape analysis. Our approach improves robustness against unforeseen noise corruptions by 4.2-18.4 % over adversarial training and other strong diverse data augmentation baselines across several benchmarks. Furthermore, it improves robustness and uncertainty calibration by 3.7% and 5.5 %, respectively, against all common corruptions (weather, digital, blur, noise) when combined with state-of-the-art diverse data augmentations. Code is available at https://github.com/TheoT1/DiGN.
BibTeX
@inproceedings{tsiligkaridis2023diverse,
  author = {Theodoros Tsiligkaridis and Athanasios Tsiligkaridis},
  title = {{Diverse Gaussian Noise Consistency Regularization for Robustness and Uncertainty Calibration}},
  booktitle = {IJCNN},
  year = {2023},
  url = {https://openreview.net/forum?id=EQ3TZfkfLu},
}
Figure