Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck

Junho Kim, Byung-Kwan Lee, Yong Man Ro

TL;DR

Researchers from KAIST have developed a method to explicitly separate "robust" features (stable patterns) from "non-robust" features (brittle, predictive patterns) in neural networks using the Information Bottleneck principle. Their work reveals that non-robust features are not just random noise but contain human-perceptible semantic information that adversarial attacks exploit to flip model predictions.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are notoriously vulnerable to adversarial examples—images with imperceptible perturbations that cause a model to misclassify them. A leading hypothesis suggests that datasets contain two types of features:

  1. Robust features: Patterns that are stable and remain predictive even under small perturbations.
  2. Non-robust features: Patterns that are highly predictive for classification but are "brittle" and easily manipulated by noise.

While this theory is widely accepted, actually "seeing" and separating these features in a trained model has been difficult. This paper addresses the gap by providing a mathematical framework to distill these features in the intermediate layers of a network, allowing researchers to study how they interact and contribute to model failure.

Methods

The authors utilize the Information Bottleneck (IB) framework to regulate information flow in the feature space.

Information Bottleneck architecture for distilling robust and non-robust features.
  1. Noise Injection: They inject Gaussian noise into specific feature units in the intermediate layers. By optimizing the magnitude of this noise ($\sigma$), they can measure "prediction sensitivity."
  2. The Bottleneck Objective: They minimize a loss function that balances two goals: keeping the features predictive of the target label (minimizing cross-entropy) while compressing the information from the input image (minimizing information loss).
  3. Dichotomization:
    • Robust Features ($Z_r$): Feature units that can tolerate large amounts of noise while still allowing the model to predict correctly.
  4. Non-robust Features ($Z_{nr}$): Feature units that are highly sensitive to noise; even small variations here break the model's prediction.
  5. NRF Attack: Based on this distillation, they propose a new attack mechanism called the Non-Robust Feature attack (NRF). This attack specifically targets and amplifies the gradients of non-robust features to break model robustness more effectively than standard methods like PGD.

Datasets

The authors evaluated their approach on three standard computer vision benchmarks:

  • CIFAR-10: A dataset of 60,000 small color images in 10 classes.
  • SVHN (Street View House Numbers): Real-world images of house numbers.
  • Tiny-ImageNet: A subset of the ImageNet dataset with 200 classes at a lower resolution.

They used popular architectures like VGG-16 and WideResNet-28-10 for their experiments.

Key Insights and Ideas

  • Non-robust features are semantic: Contrary to the idea that non-robust features are just "unintelligible statistical noise," the authors used feature visualization to show they actually contain human-perceptible shapes and patterns. For example, a non-robust feature for a "Bird" might look like a blurry wing or beak.
  • The "Aha" Moment on Misclassification: When an adversarial attack succeeds (e.g., turning a "Bird" into a "Deer"), the visualization of the non-robust features in the attacked image actually looks like the target class ("Deer"). This proves that attacks work by subtly shifting these brittle features to mimic a different category.
  • Robustness is Invariant: Robust features ($Z_r$) maintain high classification accuracy even when the image is under heavy adversarial attack, confirming they are the "true" anchors of the model's intelligence.
t-SNE visualizations of robust and non-robust features for clean and adversarial examples.

As seen in the t-SNE plots above, robust features ($Z_r$) maintain distinct, well-separated clusters even under PGD attack. In contrast, non-robust features ($Z_{nr}$) scatter and shift, illustrating their vulnerability to adversarial manipulation.

Key Results

  • Feature Accuracy: In CIFAR-10, while the full intermediate features achieved ~79% accuracy, the distilled Robust features achieved nearly 99% accuracy on clean images, showing they capture the core identity of the objects.
  • Attack Effectiveness: The proposed NRF attack significantly outperformed standard attacks (FGSM, PGD, CW) in breaking "robust" models. For a VGG-16 model on CIFAR-10, NRF reduced accuracy to 27.4%, whereas the standard PGD attack only brought it down to 44.7%.
  • Visual Correlation: The paper provides striking visual evidence showing that non-robust features cluster according to the adversarial label, not the true label.

Limitations and Future Work

  • Layer Specificity: The authors primarily focused on the last convolutional layer for distillation, as it contains the highest-level semantic concepts. They acknowledge that how these features interact across lower layers remains an open question.
  • Computational Overhead: Using Information Bottleneck requires an optimization step to determine the noise variation for each image, which is more computationally intensive than standard inference.
  • Future Directions: The authors suggest that this distillation method could be used to build better adversarial detection systems or to design new defense strategies that explicitly ignore non-robust features during training.

Summary updated

Abstract

Adversarial examples, generated by carefully crafted perturbation, have attracted considerable attention in research fields. Recent works have argued that the existence of the robust and non-robust features is a primary cause of the adversarial examples, and investigated their internal interactions in the feature space. In this paper, we propose a way of explicitly distilling feature representation into the robust and non-robust features, using Information Bottleneck. Specifically, we inject noise variation to each feature unit and evaluate the information flow in the feature representation to dichotomize feature units either robust or non-robust, based on the noise variation magnitude. Through comprehensive experiments, we demonstrate that the distilled features are highly correlated with adversarial prediction, and they have human-perceptible semantic information by themselves. Furthermore, we present an attack mechanism intensifying the gradient of non-robust features that is directly related to the model prediction, and validate its effectiveness of breaking model robustness.
BibTeX
@inproceedings{kim2021distilling,
  author = {Junho Kim and Byung-Kwan Lee and Yong Man Ro},
  title = {{Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=UdzUobLwX2i},
}
Figure