Encoding Robustness to Image Style via Adversarial Feature Perturbations

Manli Shu, Zuxuan Wu, Micah Goldblum, Tom Goldstein

Abstract

Adversarial training is the industry standard for producing models that are robust to small adversarial perturbations. However, machine learning practitioners need models that are robust to other kinds of changes that occur naturally, such as changes in the style or illumination of input images. Such changes in input distribution have been effectively modeled as shifts in the mean and variance of deep image features. We adapt adversarial training by directly perturbing feature statistics, rather than image pixels, to produce models that are robust to various unseen distributional shifts. We explore the relationship between these perturbations and distributional shifts by visualizing adversarial features. Our proposed method, Adversarial Batch Normalization (AdvBN), is a single network layer that generates worst-case feature perturbations during training. By fine-tuning neural networks on adversarial feature distributions, we observe improved robustness of networks to various unseen distributional shifts, including style variations and image corruptions. In addition, we show that our proposed adversarial feature perturbation can be complementary to existing image space data augmentation methods, leading to improved performance.

Introduction

Deep learning models are notoriously brittle when faced with natural distribution shifts, such as rain, artistic filters, or sketches. While standard adversarial training focuses on pixel-level noise to prevent malicious attacks, it offers limited protection against these "natural" variations. Conversely, traditional data augmentation cannot realistically cover the infinite space of possible environmental conditions.

Original input image used as a baseline for feature perturbation analysis.

This research addresses the gap by targeting the "style" of an image directly within the network's internal representations. In computer vision, style is often encoded in the mean and variance of deep features. By perturbing these statistics, the proposed method forces the model to become "style-blind" and focus on core semantic content.

Methodology: Adversarial Batch Normalization (AdvBN)

The core innovation is the AdvBN layer, which implements a min-max optimization strategy. An adversary identifies shifts in feature mean ($\mu$) and variance ($\sigma$) that maximize the network's loss, while the network updates its weights to remain robust to these "worst-case" styles.

The adversarial features are generated using the following transformation: $$f_{adv} = \delta_\sigma \cdot (f - \mu(f)) + \delta_\mu \cdot \mu(f)$$ where $\delta_\mu$ and $\delta_\sigma$ are perturbations calculated via Projected Gradient Descent (PGD). To maintain performance on standard data, the architecture employs Auxiliary Batch Normalization, maintaining separate statistics for clean and adversarial features. This allows for efficient fine-tuning of pre-trained models (typically ~20 epochs) rather than training from scratch.

Experimental Evaluation

The method was evaluated across several out-of-distribution benchmarks, including ImageNet-C (common corruptions), ImageNet-Sketch, and Synthia (synthetic video for semantic segmentation).

Key Results

AdvBN consistently improves robustness metrics across diverse tasks:

  • Corruption Robustness: On ResNet-50, AdvBN reduced the Mean Corruption Error (mCE) on ImageNet-C from 76.7 to 72.7. Combining it with AugMix further improved this to 64.6.
  • Style Generalization: Top-1 accuracy on Stylized ImageNet rose from 7.4% to 11.9% with AdvBN alone.
  • Complementarity: A major finding was that AdvBN is "orthogonal" to image-space augmentations like CutMix or AugMix, providing a cumulative performance boost when used together.

Key Insights and Visualizations

The paper builds on the concept that style is encoded in feature statistics. By passing the adversarially shifted features through a decoder, the researchers can visualize what the network perceives as a difficult or "worst-case" style.

Visualization of adversarial feature perturbations showing significant texture loss and distortion.

These visualizations reveal that adversarial attacks often target texture (e.g., removing identifying patterns) or introduce unnatural hues. AdvBN-trained models show significantly lower feature divergence between domains, effectively aligning the internal representations of disparate styles like sketches and photos.

Conclusion

AdvBN provides a computationally efficient way to encode style robustness without requiring external stylized datasets. While it introduces a minor trade-off in clean image accuracy, its ability to stack with existing augmentations makes it a powerful tool for domain generalization. Future research may extend these feature-space perturbations to non-visual domains like NLP where "style" is harder to define in the input space.

Summary updated

Abstract

Adversarial training is the industry standard for producing models that are robust to small adversarial perturbations. However, machine learning practitioners need models that are robust to other kinds of changes that occur naturally, such as changes in the style or illumination of input images. Such changes in input distribution have been effectively modeled as shifts in the mean and variance of deep image features. We adapt adversarial training by directly perturbing feature statistics, rather than image pixels, to produce models that are robust to various unseen distributional shifts. We explore the relationship between these perturbations and distributional shifts by visualizing adversarial features. Our proposed method, Adversarial Batch Normalization (AdvBN), is a single network layer that generates worst-case feature perturbations during training. By fine-tuning neural networks on adversarial feature distributions, we observe improved robustness of networks to various unseen distributional shifts, including style variations and image corruptions. In addition, we show that our proposed adversarial feature perturbation can be complementary to existing image space data augmentation methods, leading to improved performance. The source code and pre-trained models are released at \url{https://github.com/azshue/AdvBN}.
BibTeX
@inproceedings{shu2021encoding,
  author = {Manli Shu and Zuxuan Wu and Micah Goldblum and Tom Goldstein},
  title = {{Encoding Robustness to Image Style via Adversarial Feature Perturbations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=qtxl-4l2iWG},
}
Figure