Improving robustness to corruptions with multiplicative weight perturbations

Trung Trinh, Markus Heinonen, Luigi Acerbi, Samuel Kaski

TL;DR

This paper introduces Data Augmentation via Multiplicative Perturbations (DAMP), a novel training method that significantly improves deep neural network (DNN) robustness against a wide variety of unforeseen image corruptions. By injecting random multiplicative noise directly into the network's weights during training, DAMP effectively simulates input-space corruptions without sacrificing accuracy on clean images or increasing training time relative to standard SGD.

What Problem Does This Paper Address?

While deep neural networks achieve outstanding accuracy on pristine datasets, their performance often plummets when faced with real-world image corruptions like camera noise, blur, weather conditions, or compression artifacts.

The standard remedy—incorporating specific corruptions into the training data augmentation pipeline—presents a double-edged sword: it often degrades performance on clean images and fails to generalize to entirely new, unseen types of distortions. Alternative approaches like deep ensembles offer a robustness boost but incur linear increases in training and inference costs, making them impractical for large-scale models. The authors tackle this fundamental trade-off, seeking a method that delivers broad corruption robustness cheaply and without compromising clean accuracy.

Methods

The core idea behind DAMP stems from a simple but powerful mathematical intuition: input corruptions can be modeled as multiplicative perturbations in the weight space.

For a single neuron computing $z = \mathbf{w}^\top(\mathbf{x} + \boldsymbol{\epsilon})$, where $\boldsymbol{\epsilon}$ represents an input shift (corruption), one can always find an equivalent multiplicative weight perturbation (MWP) $\boldsymbol{\xi}$ such that: $$z = (\mathbf{w} \circ \boldsymbol{\xi})^\top \mathbf{x}, \quad \text{where } \boldsymbol{\xi} = 1 + \boldsymbol{\epsilon}/\mathbf{x}$$

Figure 1: Depictions of a pre-activation neuron z = w⊤x in the presence of (a) covariate shift ϵ, (b) a multiplicative weight perturbation (MWP) equivalent to ϵ, and (c) random MWPs ξ. ◦ denotes the Hadamard product. Figs. (a) and (b) show that for a covariate shift ϵ, one can always find an equivalent MWP. From this intuition, we propose to inject random MWPs ξ to the forward pass during training as shown in Fig. (c) to robustify a DNN to covariate shift.

Building on this, DAMP injects random isotropic Gaussian multiplicative noise ($\boldsymbol{\xi} \sim \mathcal{N}(\mathbf{1}, \sigma^2\mathbf{I})$) into the weights during the forward pass of training.

To prevent the computational overhead typically associated with Monte Carlo sampling, DAMP employs a clever parallelization trick: it splits each training mini-batch into $M$ equal sub-batches, applies a different random noise sample to each sub-batch in parallel, and averages the resulting gradients. This keeps the training cost identical to standard SGD.

Additionally, the paper establishes a theoretical bridge to adversarial perturbations, proving that the recently proposed Adaptive Sharpness-Aware Minimization (ASAM) algorithm implicitly optimizes DNNs under adversarial multiplicative weight perturbations.

Datasets

The authors rigorously evaluated their approach using several standard image classification datasets and an extensive suite of corruption benchmarks:

  • Base Datasets: CIFAR-10, CIFAR-100, TinyImageNet, and ImageNet.
  • Robustness Benchmarks: CIFAR-10/100-C, TinyImageNet-C, ImageNet-C, ImageNet-$\bar{\text{C}}$ (unseen corruptions), ImageNet-D (diffusion-modified backgrounds/textures), ImageNet-A (natural adversarial examples), ImageNet-Sketch, ImageNet-Drawing, ImageNet-Cartoon, and ImageNet-Hard.
  • Adversarial Evaluation: Fast Gradient Sign Method (FGSM) examples.

Key Insights and Ideas

  • Weight Space vs. Input Space: Moving the problem of simulating corruptions from the input space to the weight space allows the network to build a more generalized resilience. Instead of learning to ignore specific pixel-level patterns (like rain or compression lines), the network learns to be robust to internal feature variations.
  • Multiplicative > Additive: The authors compared DAMP against an additive counterpart (DAAP). Multiplicative perturbations proved vastly superior at improving corruption robustness, likely because they scale naturally with the magnitude of the weights, respecting the network's underlying scaling properties.
  • ViT Sample Efficiency: Vision Transformers (ViTs) notoriously require massive datasets or heavy data augmentation to match ResNet performance when trained from scratch. DAMP acts as a powerful regularizer that enables training ViTs effectively from scratch using only basic preprocessing.

Key Results

  • Superior Robustness at Half the Cost: Across CIFAR-10, CIFAR-100, and TinyImageNet, DAMP consistently outperformed standard Dropout and Sharpness-Aware Minimization (SAM) on corrupted images across various intensity levels, despite requiring only half the training compute of SAM.
Figure 4: DAMP surpasses SAM on corrupted images in most cases, despite requiring only half the training cost. We report the predictive errors (lower is better) averaged over 5 seeds. A severity level of 0 indicates no corruption. We use the same number of epochs for all methods.
  • ImageNet Performance: When training a ResNet50 on ImageNet, DAMP achieved a lower average corrupted error (74.2%) compared to Dropout (76.6%) and SAM (75.8%).
  • ViT Breakthrough: Remarkably, DAMP trained a ViT-S/16 from scratch on ImageNet to a top-1 clean error of 23.7% using only basic Inception-style preprocessing. This significantly beats the Dropout baseline (28.74%) and matches a standard ResNet50, demonstrating that DAMP can substitute for complex, hand-engineered data augmentation pipelines.

Limitations and Future Work

While highly effective, the authors note a couple of limitations:

  1. Theoretical Simplifications: The formal proof bounding the training loss under corruption assumes a simple feedforward neural network. It does not explicitly account for modern architectural components like batch normalization, layer normalization, or self-attention mechanisms.
  2. Noise Distribution Limits: The empirical work focused exclusively on Gaussian multiplicative noise. Exploring more sophisticated or structured noise distributions could potentially unlock even higher levels of robustness.

Future work includes extending DAMP beyond computer vision into domains like natural language processing (NLP) and reinforcement learning.

Summary updated

Abstract

Deep neural networks (DNNs) excel on clean images but struggle with corrupted ones. Incorporating specific corruptions into the data augmentation pipeline can improve robustness to those corruptions but may harm performance on clean images and other types of distortion. In this paper, we introduce an alternative approach that improves the robustness of DNNs to a wide range of corruptions without compromising accuracy on clean images. We first demonstrate that input perturbations can be mimicked by multiplicative perturbations in the weight space. Leveraging this, we propose Data Augmentation via Multiplicative Perturbation (DAMP), a training method that optimizes DNNs under random multiplicative weight perturbations. We also examine the recently proposed Adaptive Sharpness-Aware Minimization (ASAM) and show that it optimizes DNNs under adversarial multiplicative weight perturbations. Experiments on image classification datasets (CIFAR-10/100, TinyImageNet and ImageNet) and neural network architectures (ResNet50, ViT-S/16, ViT-B/16) show that DAMP enhances model generalization performance in the presence of corruptions across different settings. Notably, DAMP is able to train a ViT-S/16 on ImageNet from scratch, reaching the top-1 error of 23.7% which is comparable to ResNet50 without extensive data augmentations.
BibTeX
@inproceedings{trinh2024improving,
  author = {Trung Trinh and Markus Heinonen and Luigi Acerbi and Samuel Kaski},
  title = {{Improving robustness to corruptions with multiplicative weight perturbations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=M8dy0ZuSb1},
}
Figure