Unlabeled Data Improves Adversarial Robustness

Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C. Duchi, Percy Liang

Introduction

Adversarial examples—small, often imperceptible perturbations that cause machine learning models to fail—are a major obstacle to the deployment of reliable AI. While it is well-established that training for robustness is "data-hungry," the high cost of gathering quality labels remains a bottleneck. This research demonstrates that unlabeled data can effectively bridge the "sample complexity gap" between standard and robust classification. By introducing Robust Self-Training (RST), the authors show that models can achieve state-of-the-art robustness by learning from the distribution of natural inputs, even without explicit labels for every sample.

Methodology: Robust Self-Training (RST)

The proposed RST framework utilizes unlabeled data to enforce stability around natural inputs. The process consists of two stages:

  1. Pseudo-label Generation: A standard model is trained on available labeled data and used to generate labels for a large unlabeled pool.
  2. Robust Training: A final model is trained using a robust objective (e.g., TRADES for $\ell_\infty$ robustness or Stability Training for certified $\ell_2$ robustness) on the combined labeled and pseudo-labeled dataset.

The core insight is that robustness is fundamentally about stability. Since stability is a property of the model's behavior around inputs, knowing the input distribution via unlabeled data is nearly as valuable as having ground-truth labels.

Experimental Setup

The authors evaluated RST on CIFAR-10 and SVHN. For CIFAR-10, they augmented the standard 50K labeled images with 500K unlabeled images mined from the "80 Million Tiny Images" dataset. For SVHN, they treated the "extra" 531K images as unlabeled data by stripping their labels to test the theory that labels are not the primary driver of robustness gains.

Key Results

RST achieved significant improvements across multiple robustness benchmarks. On CIFAR-10, it reached a new state-of-the-art robust accuracy of 62.5% against strong $\ell_\infty$ attacks, a 7% improvement over the previous best method, TRADES.

Robust accuracy of RST against various adversarial attacks on CIFAR-10.

Furthermore, RST significantly advanced the field of certified robustness. In the $\ell_2$ setting, certified accuracy improved by 10% over previous benchmarks. Notably, RST also helps mitigate the typical trade-off between robustness and standard accuracy; in some certified settings, standard accuracy improved by over 16% compared to baselines.

Certified $\ell_2$ robustness (a) and $\ell_\infty$ accuracy at $\epsilon=2/255$ (b) for RST.

On the SVHN dataset, the use of pseudo-labels yielded performance within 1% of that achieved using true labels, confirming that the primary benefit of the additional data comes from exposing the model to the input distribution rather than the labels themselves.

Discussion and Conclusion

The success of RST suggests that the increased sample complexity required for robustness can be satisfied by leveraging the abundance of unlabeled data available in the wild. However, the method is not without challenges: training on large-scale unlabeled sets is computationally intensive, and the relevance of the unlabeled data to the target task is crucial for performance. Future work may explore scaling these techniques to more complex datasets like ImageNet and refining data selection processes to maximize the utility of unlabeled pools.

Summary updated

Abstract

We demonstrate, theoretically and empirically, that adversarial robustness can significantly benefit from semisupervised learning. Theoretically, we revisit the simple Gaussian model of Schmidt et al. that shows a sample complexity gap between standard and robust classification. We prove that unlabeled data bridges this gap: a simple semisupervised learning procedure (self-training) achieves high robust accuracy using the same number of labels required for achieving high standard accuracy. Empirically, we augment CIFAR-10 with 500K unlabeled images sourced from 80 Million Tiny Images and use robust self-training to outperform state-of-the-art robust accuracies by over 5 points in (i) $\ell_\infty$ robustness against several strong attacks via adversarial training and (ii) certified $\ell_2$ and $\ell_\infty$ robustness via randomized smoothing. On SVHN, adding the dataset's own extra training set with the labels removed provides gains of 4 to 10 points, within 1 point of the gain from using the extra labels.
BibTeX
@inproceedings{carmon2019unlabeled,
  author = {Yair Carmon and Aditi Raghunathan and Ludwig Schmidt and John C. Duchi and Percy Liang},
  title = {{Unlabeled Data Improves Adversarial Robustness}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=8v2ZAUNZss},
}
Figure