Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization

Sicheng Zhu, Xiao Zhang, David Evans

Abstract

Training machine learning models that are robust against adversarial inputs poses seemingly insurmountable challenges. To better understand adversarial robustness, we consider the underlying problem of learning robust representations. We develop a notion of representation vulnerability that captures the maximum change of mutual information between the input and output distributions, under the worst-case input perturbation. Then, we prove a theorem that establishes a lower bound on the minimum adversarial risk that can be achieved for any downstream classifier based on its representation vulnerability. We propose an unsupervised learning method for obtaining intrinsically robust representations by maximizing the worst-case mutual information between the input and output distributions. Experiments on downstream classification tasks support the robustness of the representations found using unsupervised learning with our training principle.

Introduction

Deep learning models are notoriously fragile; tiny, human-imperceptible "adversarial" changes to an input can trick a state-of-the-art model into misclassification. While most existing defenses focus on adversarial training for specific classifiers, this paper shifts the focus to the representation layer. The authors argue that if the features themselves are "vulnerable"—changing drastically when the input is slightly nudged—then no downstream classifier can ever be truly robust. The challenge lies in defining and learning these robust features without relying on labels (unsupervised learning), allowing them to be reused across various tasks.

Methodology: Worst-Case Mutual Information Maximization

The authors introduce Representation Vulnerability, a metric measuring the maximum drop in Mutual Information (MI) between the input and the representation when the input distribution is perturbed within an $\infty$-Wasserstein ball. To learn robust representations, they propose Worst-Case Mutual Information Maximization. Instead of standard unsupervised learning that maximizes MI between an image and its features, this principle maximizes the information that remains even under the worst possible attack.

Key technical components include:

  • MINE (Mutual Information Neural Estimation): A neural network-based tool used to estimate complex MI values.
  • Min-Max Optimization: A training loop where one part of the system finds the most damaging perturbation (minimizing MI) while the encoder produces features that keep MI high (maximizing the minimum).
  • Heuristic Search: Utilizing projected gradient descent (PGD) to find strong empirical perturbations during training.

A fundamental contribution of this work is proving that if a representation is vulnerable (low worst-case MI), there is a mathematical "speed limit" on the robustness of any downstream classifier. This theoretical link suggests that representation vulnerability is a strong predictor of empirical adversarial risk.

Figure 1: Correlation between representation vulnerability and the accuracy gap (Natural - Adversarial).

As shown in Figure 1, there is a clear positive correlation between representation vulnerability and the performance drop under attack. Models with lower vulnerability consistently exhibit a smaller gap between natural and adversarial accuracy, validating the theoretical foundation.

Key Results

The proposed method was evaluated on standard benchmarks including MNIST, Fashion-MNIST, SVHN, and CIFAR-10. The results demonstrate that robust representations can be learned effectively without labels.

Figure 4: Comparison of natural and adversarial accuracy across different representation learning methods on CIFAR-10.

As detailed in Figure 4, on the CIFAR-10 dataset, a downstream MLP classifier using these robust representations achieved 31.52% adversarial accuracy, a massive improvement over the 0.22% achieved by standard unsupervised representations (Hjelm et al., 2018). Furthermore, these robust representations often improved "natural" (non-attacked) accuracy, reaching 62.54% compared to the baseline's 58.77%.

Discussion and Future Work

While this method significantly outperforms standard unsupervised learning, a gap remains compared to fully-supervised adversarial training (which reaches ~40% adversarial accuracy on CIFAR-10). Additionally, the min-max optimization is computationally expensive and relies on heuristic searches for perturbations. Future research could explore scaling this approach to larger datasets like ImageNet and investigating the transferability of these robust features to tasks such as object detection or segmentation.

Summary updated

Abstract

Training machine learning models that are robust against adversarial inputs poses seemingly insurmountable challenges. To better understand adversarial robustness, we consider the underlying problem of learning robust representations. We develop a notion of representation vulnerability that captures the maximum change of mutual information between the input and output distributions, under the worst-case input perturbation. Then, we prove a theorem that establishes a lower bound on the minimum adversarial risk that can be achieved for any downstream classifier based on its representation vulnerability. We propose an unsupervised learning method for obtaining intrinsically robust representations by maximizing the worst-case mutual information between the input and output distributions. Experiments on downstream classification tasks support the robustness of the representations found using unsupervised learning with our training principle.
BibTeX
@inproceedings{zhu2020learning,
  author = {Sicheng Zhu and Xiao Zhang and David Evans},
  title = {{Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=5-hHA9yqQmT},
}
Figure