How to Craft Backdoors with Unlabeled Data Alone?

Yifei Wang, Wenhan Ma, Stefanie Jegelka, Yisen Wang

Abstract

Relying only on unlabeled data, Self-supervised learning (SSL) can learn rich features in an economical and scalable way. As the drive-horse for building foundation models, SSL has received a lot of attention recently with wide applications, which also raises security concerns where backdoor attack is a major type of threat: if the released dataset is maliciously poisoned, backdoored SSL models can behave badly when triggers are injected to test samples. The goal of this work is to investigate this potential risk. We notice that existing backdoors all require a considerable amount of labeled data that may not be available for SSL. To circumvent this limitation, we explore a more restrictive setting called no-label backdoors, where we only have access to the unlabeled data alone, where the key challenge is how to select the proper poison set without using label information. We propose two strategies for poison selection: clustering-based selection using pseudolabels, and contrastive selection derived from the mutual information principle. Experiments on CIFAR-10 and ImageNet-100 show that both no-label backdoors are effective on many SSL methods and outperform random poisoning by a large margin.

Introduction

The rise of foundation models relies heavily on Self-Supervised Learning (SSL), where models are trained on massive, unlabeled datasets scraped from the internet. While efficient, this creates a significant security surface. Traditional backdoor attacks are "label-aware"—the attacker knows the labels and injects a trigger into a specific target class. However, in the SSL pipeline, an attacker often only has access to a raw, unlabeled pool of data. Without labels, it is difficult to select a "consistent" set of images to poison. This paper explores the No-Label Backdoor (NLB) setting, demonstrating how to craft effective backdoors when labels are completely unavailable.

The NLB Threat Model

In the NLB scenario, the attacker's goal is to force the SSL model to associate a trigger with a specific (albeit unlabeled) group of images. If an attacker poisons random images, the model fails to learn a strong correlation between the trigger and any specific feature, rendering the attack ineffective. The authors frame a backdoor as a synthetic feature being introduced into the dataset; for the model to prioritize this feature, the underlying images must be semantically consistent.

Overview of the No-Label Backdoor (NLB) attack pipeline.

Methodology

The authors propose two primary strategies for selecting a poison set from unlabeled data:

  1. Clustering-based NLB: This method uses a pre-trained SSL encoder to extract features from the unlabeled data and then applies K-means clustering. Each cluster acts as a "pseudoclass." The attacker selects the smallest cluster that meets their poisoning budget, assuming that smaller clusters are more likely to be semantically consistent (high Cluster Consistency Rate).
  2. Contrastive NLB (Total Contrastive Similarity - TCS): Recognizing that clustering can be unstable due to random initialization, the authors derive a more principled approach based on Mutual Information (MI). They argue that a good backdoor should be "easy to learn," meaning the "backdoor feature" should have high mutual information with the model's learning objective. Practically, the attacker chooses a subset of samples that have high similarity to each other but low similarity to the rest of the dataset.

Experimental Evaluation

The researchers evaluated their attacks on two standard computer vision benchmarks: CIFAR-10 (60,000 images, 10 classes) and ImageNet-100 (a 100-class subset of ImageNet-1k). In both cases, the data was split into an unlabeled pre-training set and a labeled downstream transfer set to simulate real-world deployment.

Key Results

  • Attack Success Rate (ASR): On CIFAR-10, Contrastive NLB achieved an ASR of 88.21% (using SimCLR), compared to only ~15% for random poisoning. On ImageNet-100, it reached 74.46%, while random poisoning effectively failed at ~1%.
  • Efficiency: The selection process is extremely fast. Using a single GPU, the poison set selection for CIFAR-10 takes roughly 10 seconds.
  • Stealthiness: The attacks do not significantly degrade the model's performance on clean data, making the backdoor difficult to detect during standard validation.
Performance metrics for NLB, showing the effectiveness of clustering and contrastive selection over random poisoning.

Discussion and Insights

The experiments reveal that No-Label Backdoors, specifically the contrastive version, perform almost as well as an "Oracle" attack with full access to ground-truth labels. This suggests that label information is not a prerequisite for high-potency attacks in SSL. Furthermore, these backdoors are surprisingly resilient to common defenses like fine-tuning; even after the model is "cleaned" with a small amount of labeled data, the backdoor often persists.

Limitations and Future Work

A current limitation is the requirement of a pre-trained SSL encoder to extract features for selection. While these are widely available online (e.g., on HuggingFace), an attacker without any starting model might face a "cold start" problem. Additionally, because the attacker doesn't use labels, they cannot easily choose a specific semantic target. Future work is needed to determine if these principles apply to other modalities, such as Large Language Models (LLMs).

Summary updated

Abstract

Relying only on unlabeled data, Self-supervised learning (SSL) can learn rich features in an economical and scalable way. As the drive-horse for building foundation models, SSL has received a lot of attention recently with wide applications, which also raises security concerns where backdoor attack is a major type of threat: if the released dataset is maliciously poisoned, backdoored SSL models can behave badly when triggers are injected to test samples. The goal of this work is to investigate this potential risk. We notice that existing backdoors all require a considerable amount of \emph{labeled} data that may not be available for SSL. To circumvent this limitation, we explore a more restrictive setting called no-label backdoors, where we only have access to the unlabeled data alone, where the key challenge is how to select the proper poison set without using label information. We propose two strategies for poison selection: clustering-based selection using pseudolabels, and contrastive selection derived from the mutual information principle. Experiments on CIFAR-10 and ImageNet-100 show that both no-label backdoors are effective on many SSL methods and outperform random poisoning by a large margin. Code will be available at https://github.com/PKU-ML/nlb.
BibTeX
@misc{wang2024how,
  author = {Yifei Wang and Wenhan Ma and Stefanie Jegelka and Yisen Wang},
  title = {{How to Craft Backdoors with Unlabeled Data Alone?}},
  year = {2024},
  eprint = {2404.06694},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.06694},
}
Figure