TL;DR
Self-supervised learning (SSL) trains foundation models on massive unlabeled datasets, but what if an attacker poisons the data without knowing any ground-truth labels? This paper introduces No-Label Backdoors (NLB), showing that attackers can reliably inject backdoors into SSL models using only unlabeled data. By framing poison selection through a mutual information lens, the proposed contrastive selection strategy achieves attack success rates rivaling attacks that rely on full label supervision.
What Problem Does This Paper Address?
Backdoor attacks traditionally rely on label awareness: an attacker embeds a trigger pattern (e.g., a small watermark) into images belonging to a chosen "target class," teaching the model to associate the trigger with that specific class. However, the core appeal of self-supervised learning is that massive training sets (e.g., Common Crawl, medical microscopy corpora) are completely unannotated.
If a malicious actor hosts an unauthorized mirror or modified copy of a public unlabeled dataset, how could they backdoor an SSL pipeline without knowing ground-truth labels? Previous SSL backdoor methods assumed access to substantial label information, leaving a blind spot: can effective backdoors be injected using unlabeled data alone?
Methods
The paper formulates this problem as an untargeted or pseudo-targeted attack: the attacker injects triggers into a small budget ($M$ samples) of the unlabeled corpus, aiming to cluster poisoned test inputs into an emergent "pseudo-target" class while preserving clean data performance.
The authors propose two strategies for selecting which samples to poison:
- Clustering-based NLB (Indirect Selection): Uses a pre-trained SSL encoder to extract representations, runs K-means, and selects the smallest cluster whose size exceeds the budget $M$. While intuitive, K-means is sensitive to random initialization and can yield low class consistency.
- Contrastive NLB via Mutual Information (Direct Selection): The authors treat the backdoor trigger indicator $S \in \{0, 1\}$ as a new feature and maximize the mutual information $I(X; S)$ between inputs and the trigger. They derive a tractable variational lower bound called Total Contrastive Similarity (TCS): $$\mathcal{L}_{\text{TCS}}(P) = \sum_{x \in P} \sum_{x^+ \in P} f(x)^\top f(x^+) - \sum_{x \in P} \sum_{x' \in \mathcal{N}_x} f(x)^\top f(x')$$ In practice, an anchor sample and its $M$ nearest neighbors are chosen to maximize within-group similarity (positives) while minimizing similarity to hard negatives outside the poison set.
Datasets
The authors evaluate their attacks on:
- CIFAR-10: Default poison rate of 6%.
- ImageNet-100: A 100-class subset of ImageNet-1k, using a stealthier 0.6% poison rate.
Pretraining was conducted across multiple prominent SSL frameworks—including SimCLR, MoCo v2, BYOL, Barlow Twins, DINO, and MoCo-v3—using ResNet-18 and vision transformer backbones.
Key Insights and Ideas
- Random poisoning fails: Simply poisoning random unlabeled images yields essentially random-guess Attack Success Rates (ASR ~10% on CIFAR-10, ~1% on ImageNet-100). Backdoors require tight semantic consistency among poisoned instances.
- Backdoors as mutual information maximization: Backdoor triggers are simply features. Framing poison selection as maximizing information gain ($I(X; S)$) mathematically explains why standard attacks require poisoning identical classes—and provides a recipe for selecting poison sets when labels are missing.
- Both positive and negative terms are critical: Ablation studies show that merely clustering similar samples (positives) fails on complex datasets like ImageNet-100 (ASR drops to 1.79%). Actively repelling hard negatives is essential to isolate the poisoned cluster in representation space.
- Near-oracle performance: Contrastive NLB performs within 1% ASR and poison accuracy of an "oracle" attacker who has access to ground-truth labels.
Key Results
- High Attack Success Across SSL Paradigms: On CIFAR-10 using SimCLR, contrastive NLB achieves an 88.21% ASR (reducing triggered classification accuracy from 86.43% down to 19.59%), compared to just 14.93% ASR for random poisoning.
- Scalability to ImageNet-100: Contrastive NLB achieves 74.46% ASR on SimCLR and 73.85% ASR on Barlow Twins with only a 0.6% poison injection rate.
- Computational Efficiency: Generating the poison set takes only ~10 seconds on a single NVIDIA RTX 3090 GPU once features are extracted.
- Resistance to Fine-Tuning: When backdoored models are fully fine-tuned on downstream labeled tasks (a common defense), contrastive NLB still maintains 70.36% ASR on CIFAR-10 and 24.57% on ImageNet-100.
Limitations and Future Work
- Reliance on a Pretrained Feature Extractor: Both clustering and contrastive selection require an initial feature encoder to compute embeddings (though publicly pretrained models are widely available).
- Vulnerability to Specialized SSL Defenses: While moderately resistant to standard fine-tuning, the attack’s ASR drops sharply (to ~2.4%) when exposed to defense mechanisms specifically tailored for SSL backdoors, such as PatchSearch.
- Cross-Dataset Generalization: Transferring a backdoored encoder to an entirely different downstream label set degrades the attack from targeted misclassification to general performance degradation.
Summary updated