How to Craft Backdoors with Unlabeled Data Alone?

Yifei Wang, Wenhan Ma, Yisen WangSubmission to ICLR 2024 · Rejected

TL;DR

Self-supervised learning (SSL) trains foundation models on massive unlabeled datasets, but what if an attacker poisons the data without knowing any ground-truth labels? This paper introduces No-Label Backdoors (NLB), showing that attackers can reliably inject backdoors into SSL models using only unlabeled data. By framing poison selection through a mutual information lens, the proposed contrastive selection strategy achieves attack success rates rivaling attacks that rely on full label supervision.

What Problem Does This Paper Address?

Backdoor attacks traditionally rely on label awareness: an attacker embeds a trigger pattern (e.g., a small watermark) into images belonging to a chosen "target class," teaching the model to associate the trigger with that specific class. However, the core appeal of self-supervised learning is that massive training sets (e.g., Common Crawl, medical microscopy corpora) are completely unannotated.

If a malicious actor hosts an unauthorized mirror or modified copy of a public unlabeled dataset, how could they backdoor an SSL pipeline without knowing ground-truth labels? Previous SSL backdoor methods assumed access to substantial label information, leaving a blind spot: can effective backdoors be injected using unlabeled data alone?

Methods

The paper formulates this problem as an untargeted or pseudo-targeted attack: the attacker injects triggers into a small budget ($M$ samples) of the unlabeled corpus, aiming to cluster poisoned test inputs into an emergent "pseudo-target" class while preserving clean data performance.

The authors propose two strategies for selecting which samples to poison:

  1. Clustering-based NLB (Indirect Selection): Uses a pre-trained SSL encoder to extract representations, runs K-means, and selects the smallest cluster whose size exceeds the budget $M$. While intuitive, K-means is sensitive to random initialization and can yield low class consistency.
  2. Contrastive NLB via Mutual Information (Direct Selection): The authors treat the backdoor trigger indicator $S \in \{0, 1\}$ as a new feature and maximize the mutual information $I(X; S)$ between inputs and the trigger. They derive a tractable variational lower bound called Total Contrastive Similarity (TCS): $$\mathcal{L}_{\text{TCS}}(P) = \sum_{x \in P} \sum_{x^+ \in P} f(x)^\top f(x^+) - \sum_{x \in P} \sum_{x' \in \mathcal{N}_x} f(x)^\top f(x')$$ In practice, an anchor sample and its $M$ nearest neighbors are chosen to maximize within-group similarity (positives) while minimizing similarity to hard negatives outside the poison set.

Datasets

The authors evaluate their attacks on:

  • CIFAR-10: Default poison rate of 6%.
  • ImageNet-100: A 100-class subset of ImageNet-1k, using a stealthier 0.6% poison rate.

Pretraining was conducted across multiple prominent SSL frameworks—including SimCLR, MoCo v2, BYOL, Barlow Twins, DINO, and MoCo-v3—using ResNet-18 and vision transformer backbones.

Key Insights and Ideas

  • Random poisoning fails: Simply poisoning random unlabeled images yields essentially random-guess Attack Success Rates (ASR ~10% on CIFAR-10, ~1% on ImageNet-100). Backdoors require tight semantic consistency among poisoned instances.
  • Backdoors as mutual information maximization: Backdoor triggers are simply features. Framing poison selection as maximizing information gain ($I(X; S)$) mathematically explains why standard attacks require poisoning identical classes—and provides a recipe for selecting poison sets when labels are missing.
  • Both positive and negative terms are critical: Ablation studies show that merely clustering similar samples (positives) fails on complex datasets like ImageNet-100 (ASR drops to 1.79%). Actively repelling hard negatives is essential to isolate the poisoned cluster in representation space.
  • Near-oracle performance: Contrastive NLB performs within 1% ASR and poison accuracy of an "oracle" attacker who has access to ground-truth labels.

Key Results

  • High Attack Success Across SSL Paradigms: On CIFAR-10 using SimCLR, contrastive NLB achieves an 88.21% ASR (reducing triggered classification accuracy from 86.43% down to 19.59%), compared to just 14.93% ASR for random poisoning.
  • Scalability to ImageNet-100: Contrastive NLB achieves 74.46% ASR on SimCLR and 73.85% ASR on Barlow Twins with only a 0.6% poison injection rate.
  • Computational Efficiency: Generating the poison set takes only ~10 seconds on a single NVIDIA RTX 3090 GPU once features are extracted.
  • Resistance to Fine-Tuning: When backdoored models are fully fine-tuned on downstream labeled tasks (a common defense), contrastive NLB still maintains 70.36% ASR on CIFAR-10 and 24.57% on ImageNet-100.

Limitations and Future Work

  • Reliance on a Pretrained Feature Extractor: Both clustering and contrastive selection require an initial feature encoder to compute embeddings (though publicly pretrained models are widely available).
  • Vulnerability to Specialized SSL Defenses: While moderately resistant to standard fine-tuning, the attack’s ASR drops sharply (to ~2.4%) when exposed to defense mechanisms specifically tailored for SSL backdoors, such as PatchSearch.
  • Cross-Dataset Generalization: Transferring a backdoored encoder to an entirely different downstream label set degrades the attack from targeted misclassification to general performance degradation.

Summary updated

Abstract

Relying only on unlabeled data, Self-supervised learning (SSL) can learn rich features in an economical and scalable way. As the drive-horse for building foundation models, SSL has received a lot of attention recently with wide applications, which also raises security concerns where backdoor attack is a major type of threat: if the released dataset is maliciously poisoned, backdoored SSL models can behave badly when triggers are injected to test samples. The goal of this work is to investigate this potential risk. We notice that existing backdoors all require a considerable amount of *labeled* data that may not be available for SSL. To circumvent this limitation, we explore a more restrictive setting called no-label backdoors, where we only have access to the unlabeled data alone, where the key challenge is how to select the proper poison set without using label information. We propose two strategies for poison selection: clustering-based selection using pseudolabels, and contrastive selection derived from the mutual information principle. Experiments on CIFAR-10 and ImageNet-100 show that both no-label backdoors are effective on many SSL methods and outperform random poisoning by a large margin.
BibTeX
@misc{wang2024how,
  author = {Yifei Wang and Wenhan Ma and Yisen Wang},
  title = {{How to Craft Backdoors with Unlabeled Data Alone?}},
  year = {2024},
  eprint = {2404.06694},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.06694},
}

Loading related content...

Figure