Stylized Adversarial Defense

Muzammal Naseer, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, Fatih Porikli

Abstract

Deep learning models are notoriously vulnerable to adversarial perturbations and natural corruptions. Traditional Adversarial Training (AT) often results in a significant trade-off, reducing accuracy on clean images while failing to generalize to real-world distortions. This paper introduces Stylized Adversarial Training (SAT), a novel defense mechanism that utilizes style transfer to generate perceptually diverse adversarial examples. By forcing the model to resist perturbations that incorporate the style and content of different classes, SAT achieves state-of-the-art robustness against both white-box attacks and common corruptions without the typical drop in clean-image accuracy.

Introduction

Deep Learning models are fragile; invisible "adversarial" tweaks can trick a classifier into misidentifying a subject—for instance, seeing a wolf as a car. While training on these "broken" images is the standard defense, it typically introduces two major issues: a significant drop in clean-image accuracy and poor generalization to real-world "natural" corruptions like fog, snow, or motion blur.

Comparison of SAT and standard training across various image perturbations. SAT maintains correct classification where normal training (NT) fails.

The authors argue that existing defenses focus too narrowly on the classifier's decision boundary and ignore the richer perceptual information—the style and content—of the images themselves. SAT addresses this by generating adversaries that lie closer to the "natural image manifold," preventing the model from overfitting to non-natural mathematical artifacts.

Methodology

The core of the SAT framework consists of two main components: Stylized Adversary Generation and Max-Margin training.

Stylized Adversary Generation

Instead of adding simple gradient noise, SAT selects a "target" image from a different class and perturbs the original "source" image using a multi-task objective. This process minimizes three distinct losses:

  • Style Loss ($L_s$): Aligns the source image's texture and style (calculated via Gram matrices) with the target image.
  • Content Loss ($L_c$): Ensures the source image's high-level features remain consistent.
  • Cross-entropy Loss ($L_{ce}$): Standard adversarial goal of pushing the image across the classification boundary.
Overview of the Stylized Adversarial Training (SAT) framework, illustrating the style transfer module and max-margin learning objective.

Max-Margin Adversarial Training

Once stylized adversaries are generated, the model is trained using a contrastive-style loss designed to "stretch" the margins between classes in the feature space. The objective minimizes the distance between the clean image and its adversary while maximizing the distance between the adversary and the target class sample. This deep supervision is applied across multiple layers of the network to ensure robustness at every stage of feature extraction.

Experimental Evaluation

The authors evaluated SAT on several standard benchmarks, including CIFAR-10/100, SVHN, CINIC-10 (for distributional shift), and CIFAR-10-C (for common corruptions).

Key Results

  • White-Box Attacks: On CIFAR-10, SAT achieved 53.9% accuracy against Auto-Attack, outperforming standard Adversarial Training and Feature Scattering methods.
  • Natural Corruptions: SAT demonstrated a significant lead in handling real-world distortions. On the CIFAR-10-C benchmark, it achieved a mean accuracy of 89.0%, compared to 76.6% for the popular "Trades" defense.
  • Clean Accuracy: Unlike standard PGD training, SAT maintains high clean-image accuracy (~91% on CIFAR-10), effectively breaking the traditional robustness-accuracy trade-off.
  • Unconstrained Attacks: In tests against "occlusion" attacks, SAT was significantly more resilient, showing up to 620% better robustness than competing defenses in extreme cases.

Discussion and Conclusion

SAT demonstrates that incorporating perceptual diversity into adversarial training leads to more human-like, semantically meaningful features. While the approach is computationally intensive—taking roughly 32 hours to train on a Tesla V100 for the multi-step version—it offers a superior path toward models that are both accurate and robust. Future work may explore more sophisticated target selection or distillation techniques to further enhance efficiency and generalization.

Summary updated

Abstract

Deep Convolution Neural Networks (CNNs) can easily be fooled by subtle, imperceptible changes to the input images. To address this vulnerability, adversarial training creates perturbation patterns and includes them in the training set to robustify the model. In contrast to existing adversarial training methods that only use class-boundary information (e.g., using a cross-entropy loss), we propose to exploit additional information from the feature space to craft stronger adversaries that are in turn used to learn a robust model. Specifically, we use the style and content information of the target sample from another class, alongside its class-boundary information to create adversarial perturbations. We apply our proposed multi-task objective in a deeply supervised manner, extracting multi-scale feature knowledge to create maximally separating adversaries. Subsequently, we propose a max-margin adversarial training approach that minimizes the distance between source image and its adversary and maximizes the distance between the adversary and the target image. Our adversarial training approach demonstrates strong robustness compared to state-of-the-art defenses, generalizes well to naturally occurring corruptions and data distributional shifts, and retains the model's accuracy on clean examples.
BibTeX
@article{naseer2023stylized,
  author = {Muzammal Naseer and Salman H. Khan and Munawar Hayat and Fahad Shahbaz Khan and Fatih Porikli},
  title = {{Stylized Adversarial Defense}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=VyeGbMxD9s},
}
Figure