Encoding Robustness to Image Style via Adversarial Feature Perturbations

Manli Shu, Zuxuan Wu, Micah Goldblum, Tom Goldstein

Deep learning models are notoriously brittle. A classifier trained on clear, sunny photos often fails when faced with rain, fog, or even a simple Instagram filter. While Adversarial Training (AT) is the gold standard for defending against microscopic, invisible pixel changes (noise), it rarely helps with these macro-level "style" shifts.

Clean Image Example

To address this, researchers have developed Adversarial Batch Normalization (AdvBN), a method that improves a model's robustness to style shifts—like lighting, weather, or textures—by directly perturbing feature statistics during training. Unlike traditional adversarial training that focuses on pixel noise, AdvBN simulates worst-case distribution shifts, leading to significantly better generalization on out-of-distribution datasets.

The Challenge of Defining "Style"

The primary challenge in training for style robustness is that "style" is hard to define mathematically at the pixel level. However, previous research in style transfer has shown that the mean and variance of deep features effectively capture the style of an image. This paper asks a fundamental question: Can we use the machinery of adversarial training—not on pixels, but on these feature statistics—to make models style-robust?

Stylized Image Example

Adversarial Batch Normalization (AdvBN)

The authors introduce a novel network layer called AdvBN, which operates on a few key principles:

  • The Min-Max Game: During training, the model plays a game against itself. An "adversary" tries to find the most damaging shift in feature mean ($\mu$) and variance ($\sigma$) to maximize the classification loss. Simultaneously, the model updates its weights to minimize this loss.
  • PGD on Statistics: The adversary uses Projected Gradient Descent (PGD) to iteratively perturb the feature statistics within a defined bound ($\epsilon$). This effectively simulates a "worst-case" style shift for every batch of data.
  • Plug-and-Play Efficiency: AdvBN is implemented as a single layer. The authors found that they don't need to train from scratch; they can simply insert AdvBN into a pre-trained model (typically after the second convolutional stage) and fine-tune for just 20 epochs.
  • Auxiliary BNs: To prevent the model from "forgetting" how to handle clean images, they use a dual-path approach. Clean features pass through standard Batch Norm layers, while adversarially perturbed features pass through auxiliary Batch Norm layers.

Key Insights and Performance

The most profound insight from this research is that you don't need to generate complex "styled" images—which is computationally expensive—to train for style robustness. Perturbing the internal statistics of the network is a highly effective proxy.

The authors evaluated their approach on a wide battery of robustness benchmarks, including ImageNet-C (common corruptions), ImageNet-Instagram (aesthetic filters), ImageNet-Sketch, and Stylized ImageNet.

  • Synergy with Data Augmentation: AdvBN isn't a replacement for standard data augmentation like AugMix; instead, it acts as a force multiplier. On ImageNet-C, AdvBN combined with AugMix achieved a 64.6 mean Corruption Error (mCE), a significant improvement over AugMix's standalone 68.4 (lower is better).
  • Stylized ImageNet: AdvBN improved Top-1 accuracy from 7.4% to 11.9% as a standalone method. When added to AutoAugment, it boosted performance from 8.2% to 14.1%.
  • Semantic Segmentation: In cross-domain tests on the Synthia dataset (e.g., training on "Spring" and testing on "Night"), AdvBN consistently provided the best generalization, often outperforming specialized domain adaptation methods like MoEx.

By visualizing the features, the authors showed that AdvBN-trained models have much lower "feature divergence." Essentially, the internal representations of a clean image and a styled image look much more similar to an AdvBN model than to a standard one. Furthermore, they found that layer placement matters: robustness is best achieved by perturbing features at intermediate depths, as perturbing very deep layers can lead to chaotic representations that hurt performance.

Limitations and Future Directions

Like most robustness-enhancing techniques, there is a slight "robustness tax." Models trained with AdvBN may see a small drop in accuracy on perfectly clean, in-distribution data. Additionally, the strength of the adversarial perturbation ($\epsilon$) and the number of PGD steps require careful tuning to find the optimal balance between robustness and noise.

Future work could involve applying similar feature-space perturbations to other modalities, such as Natural Language Processing (NLP) or graph-based models, where style shifts are less intuitive but equally damaging.

Summary updated

Abstract

Adversarial training is the industry standard for producing models that are robust to small adversarial perturbations. However, machine learning practitioners need models that are robust to other kinds of changes that occur naturally, such as changes in the style or illumination of input images. Such changes in input distribution have been effectively modeled as shifts in the mean and variance of deep image features. We adapt adversarial training by directly perturbing feature statistics, rather than image pixels, to produce models that are robust to various unseen distributional shifts. We explore the relationship between these perturbations and distributional shifts by visualizing adversarial features. Our proposed method, Adversarial Batch Normalization (AdvBN), is a single network layer that generates worst-case feature perturbations during training. By fine-tuning neural networks on adversarial feature distributions, we observe improved robustness of networks to various unseen distributional shifts, including style variations and image corruptions. In addition, we show that our proposed adversarial feature perturbation can be complementary to existing image space data augmentation methods, leading to improved performance. The source code and pre-trained models are released at \url{https://github.com/azshue/AdvBN}.
BibTeX
@inproceedings{shu2021encoding,
  author = {Manli Shu and Zuxuan Wu and Micah Goldblum and Tom Goldstein},
  title = {{Encoding Robustness to Image Style via Adversarial Feature Perturbations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=qtxl-4l2iWG},
}
Figure