TL;DR
This paper introduces Data Augmentation via Multiplicative Perturbations (DAMP), a novel training method that significantly improves deep neural network (DNN) robustness against a wide variety of unforeseen image corruptions. By injecting random multiplicative noise directly into the network's weights during training, DAMP effectively simulates input-space corruptions without sacrificing accuracy on clean images or increasing training time relative to standard SGD.
What Problem Does This Paper Address?
While deep neural networks achieve outstanding accuracy on pristine datasets, their performance often plummets when faced with real-world image corruptions like camera noise, blur, weather conditions, or compression artifacts.
The standard remedy—incorporating specific corruptions into the training data augmentation pipeline—presents a double-edged sword: it often degrades performance on clean images and fails to generalize to entirely new, unseen types of distortions. Alternative approaches like deep ensembles offer a robustness boost but incur linear increases in training and inference costs, making them impractical for large-scale models. The authors tackle this fundamental trade-off, seeking a method that delivers broad corruption robustness cheaply and without compromising clean accuracy.
Methods
The core idea behind DAMP stems from a simple but powerful mathematical intuition: input corruptions can be modeled as multiplicative perturbations in the weight space.
For a single neuron computing $z = \mathbf{w}^\top(\mathbf{x} + \boldsymbol{\epsilon})$, where $\boldsymbol{\epsilon}$ represents an input shift (corruption), one can always find an equivalent multiplicative weight perturbation (MWP) $\boldsymbol{\xi}$ such that: $$z = (\mathbf{w} \circ \boldsymbol{\xi})^\top \mathbf{x}, \quad \text{where } \boldsymbol{\xi} = 1 + \boldsymbol{\epsilon}/\mathbf{x}$$
Building on this, DAMP injects random isotropic Gaussian multiplicative noise ($\boldsymbol{\xi} \sim \mathcal{N}(\mathbf{1}, \sigma^2\mathbf{I})$) into the weights during the forward pass of training.
To prevent the computational overhead typically associated with Monte Carlo sampling, DAMP employs a clever parallelization trick: it splits each training mini-batch into $M$ equal sub-batches, applies a different random noise sample to each sub-batch in parallel, and averages the resulting gradients. This keeps the training cost identical to standard SGD.
Additionally, the paper establishes a theoretical bridge to adversarial perturbations, proving that the recently proposed Adaptive Sharpness-Aware Minimization (ASAM) algorithm implicitly optimizes DNNs under adversarial multiplicative weight perturbations.
Datasets
The authors rigorously evaluated their approach using several standard image classification datasets and an extensive suite of corruption benchmarks:
- Base Datasets: CIFAR-10, CIFAR-100, TinyImageNet, and ImageNet.
- Robustness Benchmarks: CIFAR-10/100-C, TinyImageNet-C, ImageNet-C, ImageNet-$\bar{\text{C}}$ (unseen corruptions), ImageNet-D (diffusion-modified backgrounds/textures), ImageNet-A (natural adversarial examples), ImageNet-Sketch, ImageNet-Drawing, ImageNet-Cartoon, and ImageNet-Hard.
- Adversarial Evaluation: Fast Gradient Sign Method (FGSM) examples.
Key Insights and Ideas
- Weight Space vs. Input Space: Moving the problem of simulating corruptions from the input space to the weight space allows the network to build a more generalized resilience. Instead of learning to ignore specific pixel-level patterns (like rain or compression lines), the network learns to be robust to internal feature variations.
- Multiplicative > Additive: The authors compared DAMP against an additive counterpart (DAAP). Multiplicative perturbations proved vastly superior at improving corruption robustness, likely because they scale naturally with the magnitude of the weights, respecting the network's underlying scaling properties.
- ViT Sample Efficiency: Vision Transformers (ViTs) notoriously require massive datasets or heavy data augmentation to match ResNet performance when trained from scratch. DAMP acts as a powerful regularizer that enables training ViTs effectively from scratch using only basic preprocessing.
Key Results
- Superior Robustness at Half the Cost: Across CIFAR-10, CIFAR-100, and TinyImageNet, DAMP consistently outperformed standard Dropout and Sharpness-Aware Minimization (SAM) on corrupted images across various intensity levels, despite requiring only half the training compute of SAM.
- ImageNet Performance: When training a ResNet50 on ImageNet, DAMP achieved a lower average corrupted error (74.2%) compared to Dropout (76.6%) and SAM (75.8%).
- ViT Breakthrough: Remarkably, DAMP trained a ViT-S/16 from scratch on ImageNet to a top-1 clean error of 23.7% using only basic Inception-style preprocessing. This significantly beats the Dropout baseline (28.74%) and matches a standard ResNet50, demonstrating that DAMP can substitute for complex, hand-engineered data augmentation pipelines.
Limitations and Future Work
While highly effective, the authors note a couple of limitations:
- Theoretical Simplifications: The formal proof bounding the training loss under corruption assumes a simple feedforward neural network. It does not explicitly account for modern architectural components like batch normalization, layer normalization, or self-attention mechanisms.
- Noise Distribution Limits: The empirical work focused exclusively on Gaussian multiplicative noise. Exploring more sophisticated or structured noise distributions could potentially unlock even higher levels of robustness.
Future work includes extending DAMP beyond computer vision into domains like natural language processing (NLP) and reinforcement learning.
Summary updated