RUSH: Robust Contrastive Learning via Randomized Smoothing

Jiayu ZhouarXiv 2022 · DOI · arXiv

TL;DR

This paper introduces RUSH, a novel approach that combines contrastive learning with randomized smoothing to achieve robust image classification against adversarial attacks. RUSH significantly boosts adversarial robustness while maintaining high standard accuracy and reducing training costs compared to traditional adversarial training methods.

What Problem Does This Paper Address?

Machine learning models, particularly deep neural networks, are vulnerable to adversarial attacks. These attacks involve making subtle, imperceptible perturbations to input data (like images) that can cause the model to misclassify them. While adversarial training has been a popular defense, it's computationally expensive and can sometimes degrade standard accuracy. Self-supervised learning (SSL) methods, like contrastive learning, have shown great promise in learning powerful representations from unlabeled data, but they often inherit the robustness issues of supervised models. This paper investigates whether contrastive learning itself possesses an inherent robustness that can be leveraged, and proposes a method to enhance it without the heavy computational burden of traditional adversarial training.

Methods

The core of RUSH lies in its clever combination of two key ideas:

  • Natural Robustness in Contrastive Learning: The authors hypothesize and demonstrate that contrastive learning, by its nature of learning similar representations for similar inputs, implicitly encourages a form of robustness. This is linked to Lipschitz continuity, where a small change in input leads to a small change in output. By minimizing the distance between representations of similar image pairs, contrastive learning can naturally lead to a lower Lipschitz constant, making the learned features more robust.
  • Randomized Smoothing: This is a certified defense technique that makes a classifier robust by adding random noise to the input and averaging the predictions. It provides provable guarantees of robustness.

RUSH integrates these two by:

  1. Pre-training: Applying randomized smoothing to one of the augmented views of an image during the contrastive pre-training phase. This injects robustness into the learned feature representations from the start.
  2. Fine-tuning: Applying randomized smoothing to the input during the supervised fine-tuning stage.
  3. Inference: Using randomized smoothing (with Monte-Carlo sampling for estimation) to make predictions.

Crucially, RUSH uses a simplified inference strategy where the number of Monte-Carlo samples can be as low as 1, significantly reducing computational overhead without a substantial drop in performance.

Datasets

The paper uses standard image datasets for evaluation:

  • CIFAR-10
  • CIFAR-100
  • STL-10

They also explore transfer learning settings, such as pre-training on CIFAR-10 and fine-tuning on STL-10.

Key Insights and Ideas

  • Contrastive learning has inherent robustness: The paper reveals that contrastive learning, through its objective of minimizing feature distance for similar inputs, naturally promotes robustness. This is a significant finding as it suggests robustness can be achieved without explicit adversarial examples during pre-training.
  • RUSH is surprisingly efficient: By combining contrastive learning's natural robustness with randomized smoothing, RUSH achieves strong adversarial robustness with minimal additional computational cost, especially when using a single sample for randomized smoothing during inference.
  • Randomized smoothing benefits from robust features: The robust features learned via contrastive learning provide a better foundation for randomized smoothing, leading to improved overall robustness compared to using features from standard supervised pre-training.
  • Noise level in smoothing is not overly sensitive: The paper shows that RUSH's performance is relatively stable across a range of smoothing noise levels, making it easier to tune.

Key Results

  • State-of-the-art robust accuracy: On CIFAR-10, RUSH achieved 77.8% robust accuracy and 87.9% standard accuracy under an $\ell_\infty$-norm PGD attack with 8/255 perturbation, outperforming existing methods by a significant margin (over 15% improvement in robust accuracy).
  • Reduced training cost: RUSH incurs minimal additional computational overhead compared to standard contrastive learning (SimCLR), unlike traditional adversarial training which is very expensive.
  • Consistent performance: RUSH demonstrates strong performance across different datasets and transfer learning scenarios.
  • Effective even with minimal smoothing: Using only one sample for randomized smoothing during inference (IF strategy) still yields competitive robust accuracy, highlighting the efficiency of the approach.

Limitations and Future Work

  • Principled analysis of feature representation: The authors mention that a deeper, principled analysis of how the robust feature representation learned by RUSH boosts the smoothed classifier is left for future work. They hypothesize that the natural Lipschitz regularization in contrastive learning plays a key role.
  • Transferability to other SSL frameworks: While the paper shows RUSH can be extended to BYOL, further exploration across a wider range of SSL frameworks could be beneficial.
  • Understanding the weak association between labels and features: The paper notes that the weak association between label information and feature representation in RUSH might contribute to its robustness. Further investigation into this aspect could lead to new insights.

Summary updated

Abstract

Recently, adversarial training has been incorporated in self-supervised contrastive pre-training to augment label efficiency with exciting adversarial robustness. However, the robustness came at a cost of expensive adversarial training. In this paper, we show a surprising fact that contrastive pre-training has an interesting yet implicit connection with robustness, and such natural robustness in the pre trained representation enables us to design a powerful robust algorithm against adversarial attacks, RUSH, that combines the standard contrastive pre-training and randomized smoothing. It boosts both standard accuracy and robust accuracy, and significantly reduces training costs as compared with adversarial training. We use extensive empirical studies to show that the proposed RUSH outperforms robust classifiers from adversarial training, by a significant margin on common benchmarks (CIFAR-10, CIFAR-100, and STL-10) under first-order attacks. In particular, under $\ell_{\infty}$-norm perturbations of size 8/255 PGD attack on CIFAR-10, our model using ResNet-18 as backbone reached 77.8% robust accuracy and 87.9% standard accuracy. Our work has an improvement of over 15% in robust accuracy and a slight improvement in standard accuracy, compared to the state-of-the-arts.
BibTeX
@misc{zhou2022rush,
  author = {Jiayu Zhou},
  title = {{RUSH: Robust Contrastive Learning via Randomized Smoothing}},
  year = {2022},
  eprint = {2207.05127},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.05127},
}
Figure