Adversarial training—the process of training a model on examples intentionally perturbed to cause misclassification—is the gold standard for building secure neural networks. However, it suffers from a massive "sample complexity" problem: it typically requires significantly more labeled data than standard training to achieve the same level of generalization.
To bridge this gap, researchers have turned to unlabeled data. By using a model’s own predictions (pseudo-labels) or self-supervised objectives as a supervisory signal, models can learn to remain stable under attack without requiring exhaustive human annotation.
The Data Bottleneck in Robust Learning
Standard training only requires a model to be accurate on the data distribution it was trained on. Adversarial training (AT), however, requires the model to be accurate within an $\epsilon$-ball around every data point. This effectively expands the volume of the space the model must "cover," leading to what Runtian Zhai and Kun He describe as the "robustness gap" in Adversarially Robust Generalization Just Requires More Unlabeled Data.
The intuition is that while a small labeled dataset might define a decision boundary well enough for clean images, it is too sparse to define a boundary that resists malicious perturbations. Unlabeled data provides the necessary density. Jean-Baptiste Alayrac and Jonathan Uesato demonstrated in Are Labels Required for Improving Adversarial Robustness? that massive, uncurated datasets can actually outperform smaller, high-quality labeled sets for the specific purpose of defense.
Bootstrapping Robustness with Pseudo-Labels
The most common mechanism for leveraging unlabeled data is Robust Self-Training (RST). In this two-stage process, a standard (non-robust) model generates "pseudo-labels" for a large unlabeled corpus. These labels then serve as the ground truth for a second stage of adversarial training.
Aditi Raghunathan and Sang Michael Xie show in Understanding and Mitigating the Tradeoff between Robustness and Accuracy that this approach doesn't just improve robustness; it can simultaneously improve clean accuracy. By using unlabeled data to guide the model toward a more stable solution, the model avoids the "robustness tax" usually associated with adversarial training.
However, the quality of these pseudo-labels is a critical bottleneck. Jingfeng Zhang and Bo Han 0003 argue in Where is the Bottleneck of Adversarial Learning with Unlabeled Data? that if pseudo-labels are too noisy, the model inherits those biases. They propose Robust Co-Training (RCT), where two networks cross-verify labels to ensure only high-confidence, stable predictions are used for training.
Learning Robustness without Labels
Beyond pseudo-labeling, researchers are exploring how self-supervised learning can build "intrinsic" resistance to attacks. Instead of predicting a class, the model learns to align representations of different views of the same data.
In Robust Pre-Training by Adversarial Contrastive Learning, Ziyu Jiang and Zhangyang Wang introduce ACL, which integrates adversarial perturbations directly into a contrastive learning pipeline. By forcing the model to map an image and its adversarial counterpart to the same point in embedding space, the model learns features that are semantically invariant to noise. This allows a model to reach over 50% robust accuracy with as little as 1% of labeled data.
Similarly, Chengzhi Mao and Carl Vondrick developed TeCoA (Understanding Zero-Shot Adversarial Robustness for Large-Scale Models), which uses the text embeddings of a vision-language model like CLIP as a guide. By minimizing the distance between adversarial image features and their corresponding semantic text embeddings, they build robustness that generalizes to unseen "zero-shot" tasks.
Robustness in the Wild: Domain Adaptation
The power of unlabeled adversarial training is most evident in Unsupervised Domain Adaptation (UDA), where a model must adapt to a new "target" domain without any labels.
In DART: A Principled Approach to Adversarially Robust Unsupervised Domain Adaptation, Yunjuan Wang and Alexey Kurakin show that dynamically updated pseudo-labels can guide adversarial training on the target domain. This addresses two problems at once: distribution shift and adversarial vulnerability.
For specialized tasks like medical imaging, Lingrui Li and Ge Yang 0002 use an ensemble of standard and robust models to generate high-quality targets for source-free adaptation (Robust Source-Free Domain Adaptation for Fundus Image Segmentation). By combining the clean accuracy of a standard model with the noise-resilience of a robust one, they create a "teacher" signal that is more reliable than either model alone.
Why It Works: Semantic Stability
The recurring theme across these papers is that adversarial robustness is a form of consistency. Whether using pseudo-labels in GraphDefense or virtual adversarial perturbations in NLP (Consistency Training with Virtual Adversarial Discrete Perturbation), the goal is to ensure the model's internal representation of a concept remains unchanged by non-semantic noise. Unlabeled data provides the vast landscape of "normal" variations the model needs to see to understand what features are truly invariant.