To make a neural network robust against adversarial attacks, the standard recipe has long been Adversarial Training (AT). In its classic formulation, AT solves a minimax optimization problem:
$$\min_\theta \mathbb{E}_{(x,y)} \left[ \max_{\|\delta\| \le \epsilon} \mathcal{L}(f_\theta(x+\delta), y) \right]$$
Here, the inner maximization acts as an adversary, searching for a tiny perturbation $\delta$ that maximizes the model's classification loss $\mathcal{L}$ relative to the true label $y$. The outer minimization then updates the model parameters $\theta$ to resist this worst-case perturbation.
Notice the critical dependency in this equation: the ground-truth label $y$. To find a direction that confuses the model, the adversary must know what the correct answer is. As highlighted by Kejiang Chen et al. in Self-supervised Adversarial Training, this reliance on precise human annotations creates a severe supervised robustness bottleneck. It prevents us from building robust models in domains where labels are scarce, expensive, or entirely absent—such as web-scale uncurated datasets or target domains undergoing distribution shifts (Exploring Adversarially Robust Training for Unsupervised Domain Adaptation).
To break this bottleneck, a powerful paradigm shift has emerged: label-agnostic adversarial robustness. Instead of defining an attack by how much it alters a model's categorical output relative to a human label, these techniques define attacks by how much they disrupt the model's internal consistency, distributional smoothness, or semantic representations.
Virtual Adversarial Training and Local Smoothness
The foundational intuition for bypassing labels stems from the concept of local distributional smoothing. If a model is truly robust, its predictions should not change drastically within a small neighborhood around any input, regardless of whether we know that input's true class.
In Distributional Smoothing with Virtual Adversarial Training Takeru Miyato, Takeru Miyato et al. introduced Virtual Adversarial Training (VAT). Instead of maximizing a cross-entropy loss against a label $y$, the adversary maximizes the Kullback-Leibler (KL) divergence between the model’s prediction on the clean input $x$ and its prediction on the perturbed input $x+\delta$:
$$\max_{\delta} \text{KL}\left( f_\theta(x) \parallel f_\theta(x+\delta) \right)$$
Standard AT: [Clean Input x] ──(Adversary pushes AWAY from)──> [True Label y]
Virtual AT: [Clean Input x] ──(Adversary pushes AWAY from)──> [Model Prediction f(x)]
By minimizing this maximum divergence during training, the model smooths its own decision boundaries. The model's current belief state $f_\theta(x)$ serves as a "virtual" label. This allows adversarial regularization to be applied directly to massive quantities of unlabeled data.
Instance-Wise Contrastive Robustness
With the advent of self-supervised contrastive learning, researchers realized they could replace class labels entirely with instance identity. In standard contrastive learning, a model learns representations by pulling different augmentations of the same image (positive pairs) together in an embedding space while pushing different images (negative pairs) apart.
Minseon Kim et al. bridged this with adversarial defense in Robust Contrastive Learning (RoCL) (Adversarial Self-Supervised Contrastive Learning). In RoCL, the adversary's goal is to apply a perturbation $\delta$ to an instance $x$ such that its representation is pushed as far away as possible from the original instance's representation space. The model is then optimized to pull this "adversarial positive" back toward the original image's representation.
To make this process scalable, subsequent architectures decoupled the optimization steps. For instance, Chaoning Zhang et al. proposed DeACL (Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness), which separates standard representation learning from pseudo-supervised adversarial training to achieve massive speedups. Similarly, Sravanti Addepalli et al. introduced ProFeAT (ProFeAT: Projected Feature Adversarial Training for Self-Supervised Learning of Robust Representations), which utilizes a projection head during feature distillation to enforce robust representation matching without human labels.
This instance-wise intuition applies equally well to discrete domains like natural language processing. For example, Meng Zhao 0004 et al. utilized self-supervised contrastive learning to defend against word-substitution attacks (Self-Supervised Contrastive Learning with Adversarial Perturbations for Defending Word Substitution-based Attacks), forcing the model to map sentences containing adversarial synonyms to the same semantic space as the original text.
Semantic Anchors for Foundation Models
When scaling up to massive vision-language foundation models (like CLIP), discrete category labels are often too restrictive or entirely absent. To defend these models without destroying their zero-shot generalization capabilities, researchers use cross-modal representations as semantic anchors.
In the TeCoA framework (Understanding Zero-shot Adversarial Robustness for Large-Scale Models), Chengzhi Mao et al. leverage the text embedding of a description as an anchor for the image embedding. The adversary attempts to perturb an image $x$ to maximize the distance between its visual embedding and its corresponding text embedding.
[Text Prompt] ───> (Text Encoder) ───> [Text Embedding (Anchor)]
│
(Minimize Distance)
▼
[Perturbed Image x+δ] ─> (Image Encoder) ─> [Visual Embedding]
Because the text embedding captures a continuous, rich description of the scene rather than a single integer class label, defending against this text-guided attack yields models that remain robust across completely unseen downstream tasks. This paradigm allowed Zeyu Wang et al. to scale adversarial training to billions of parameters and uncurated web images in their AdvXL framework (Revisiting Adversarial Training at Scale).
Adapting to Unlabeled Domains
The label-agnostic paradigm is critical when a model must maintain robustness while transitioning from a labeled source domain (e.g., synthetic data) to an unlabeled target domain (e.g., real-world imagery). This task, known as Robust Unsupervised Domain Adaptation (UDA), cannot rely on target labels to generate defenses.
To resolve this, Yunjuan Wang et al. established DART (DART: A Principled Approach to Adversarially Robust Unsupervised Domain Adaptation), providing a theoretical framework for generating target-domain perturbations purely through consistency constraints. To handle the lack of ground truth, Wanqing Zhu et al. developed SRoUDA (SRoUDA: Meta Self-Training for Robust Unsupervised Domain Adaptation), which uses a meta-learning feedback loop to dynamically refine target pseudo-labels. This ensures that the perturbations generated during training are based on highly accurate model hypotheses rather than noisy guesses.
Even when partial or weak supervision is available—such as "complementary labels" indicating what an image is not—the optimization landscape can be highly unstable. To combat this, Jianan Zhou et al. demonstrated that models can be robustified by using a gradually increasing adversarial budget to stabilize training under weak supervision (Adversarial Training with Complementary Labels: On the Benefit of Gradually Informative Attacks).
By shifting the definition of an adversarial attack away from human-annotated labels and toward internal representation consistency, these approaches unlock the ability to train, scale, and evaluate robust machine learning models across completely unsupervised and open-world settings.
Key Foundational Work
Self-supervised Adversarial Training (2019) — contributed 1 concept elements to this direction.
Distributional Smoothing with Virtual Adversarial Training Takeru Miyato (2016) — contributed 1 concept elements to this direction.