Incompatibility Clustering as a Defense Against Backdoor Poisoning Attacks

Charles Jin, Melinda Sun, Martin Rinard

TL;DR

Researchers from MIT CSAIL have developed a novel defense against backdoor attacks called Incompatibility Clustering. By identifying subsets of data that fail to improve a model's performance on other subsets—a property termed "incompatibility"—their algorithm, Inverse Self-Paced Learning (ISPL), effectively isolates poisoned data from clean data, reducing attack success rates to below 1% in the vast majority of tested scenarios with minimal impact on model accuracy.

What Problem Does This Paper Address?

Deep learning models are notoriously vulnerable to backdoor poisoning attacks. In these scenarios, an attacker injects a small amount of "poisoned" data into the training set. This data contains a specific trigger (like a small pixel patch or a watermark). During deployment, the model behaves normally on clean inputs but misclassifies any input containing the trigger into a target class chosen by the attacker.

Defending against these attacks is difficult because poisoned samples often look perfectly normal to traditional outlier detection methods. Previous defenses often relied on analyzing the internal activations of a pre-trained model, but these can be bypassed by sophisticated attackers. The authors identify a fundamental gap: existing methods don't fully leverage the interaction between the dataset and the training process itself.

Methods

The authors introduce the concept of incompatibility. Two subsets of data are incompatible if training on one does not improve the model's performance on the other.

To operationalize this, they propose the Inverse Self-Paced Learning (ISPL) algorithm (Algorithm 1 in the paper). ISPL works through an iterative refinement process:

  1. Sampling: Randomly sample a small subset of the current data.
  2. Training: Train a model on this small subset.
  3. Trimming: Evaluate the model on the entire dataset and keep only the samples with the lowest loss (those most "compatible" with the training subset).
  4. Iteration: Repeat this process, gradually decreasing the size of the kept subset until it (ideally) contains only highly compatible, homogeneous data (either all clean or all poisoned).

Once the dataset is partitioned into these homogeneous clusters, the authors use a Boosting mechanism (Algorithm 2). They train "weak learners" on each cluster and have them vote on the other clusters. Since clean data is typically the majority and mutually compatible, clean clusters will vote for each other, while poisoned clusters—being incompatible with clean data—will be isolated and discarded.

Datasets

The authors evaluated their defense against three types of backdoor attacks:

  • Dirty Label Backdoor (DLBD): Simple pixel-based patches where the label is also changed.
  • WATERMARK: Blending a semi-transparent image (like a copyright sign) into the source image.
  • Clean Label Backdoor (CLBD): A more sophisticated attack where the poisoned images look like their original class to humans but contain adversarial perturbations that trick the model.

These attacks were tested on standard computer vision benchmarks:

  • CIFAR-10: A dataset of 60,000 small color images in 10 classes.
  • GTSRB (German Traffic Sign Recognition Benchmark): A real-world dataset for traffic sign classification.
  • Imagenette: A smaller, easier-to-classify subset of 10 classes from the massive ImageNet dataset (used for high-resolution testing).
Figure 1: Pairs of poisoned (top) and clean (bottom) samples for all datasets, selected for visible triggers. From left to right: CIFAR-10 with CLBD using GAN, ℓ2, and ℓ∞perturbations, DLBD, and WATERMARK attacks; and GTSRB with DLBD and WATERMARK attacks.

Key Insights and Ideas

  • Training Dynamics as a Signal: The most powerful insight is that the way a model learns from data is a better indicator of "poison" than the static features of the data itself. Poisoned data is fundamentally "selfish"—it helps the model learn the backdoor trigger but contributes nothing to learning the actual clean features of the class.
  • Self-Expansion: The authors formalize "compatibility" through a self-expansion property. A set that generalizes well to itself from a small random subset is likely homogeneous.
  • Dynamic Resizing is Key: Unlike previous "trimmed loss" methods that remove a fixed number of samples, ISPL's iterative shrinking is crucial for finding the core "compatible" subset.
  • Provable Separation: The paper provides theoretical proofs showing that for linear classifiers, backdoor attacks must satisfy this incompatibility property, allowing for provable separation.

Key Results

The proposed ISPL+B defense demonstrated state-of-the-art performance:

  • High Success Rate: It reduced the Targeted Misclassification Rate (TMR) to below 1% in 134 out of 165 tested scenarios.
  • Minimal Utility Cost: The drop in clean accuracy was only about 2% for CIFAR-10 and was negligible for GTSRB.
  • Superiority over Baselines: In a direct comparison on a standard dirty-label attack, ISPL+B successfully defended 22 out of 24 scenarios, while the next best baseline (Activation Clustering) only managed 5.
Table 1: Summary of performance for ISPL+B across several datasets and poisoning strategies. Columns display different datasets and poisoning strategies (e.g., poisoning CIFAR-10 using the CLBD attack). The rows display different amounts of poison (e.g., ϵ = 5 refers to 5% of the source class being poisoned). An entry of the form “7 / 8” indicates that ISPL+B successfully defended 7 of the 8 scenarios in the corresponding setting. Please refer to Section 4.2 for a detailed description.

The authors also showed that the incompatibility property holds empirically across various attack types. As the ratio of poisoned data increases, the model's ability to generalize from poisoned data to clean data (and vice versa) consistently drops.

Figure 2: Results from incompatibility experiments on CIFAR-10, DLBD (left) and CLBD (right). Each subplot shows how accuracy changes over different subsets of data as the amount of poison increases. The top plot displays change in accuracy on clean training data, i.e., ϵ(C|P; α)−ϵ(C; α); the next two plots separate the change in accuracy on C into source and non-source classes; the bottom plot shows change in accuracy on poisoned data P. All axes use linear scales.

Limitations and Future Work

  • Computational Overhead: ISPL requires training multiple models over several iterations, which is more computationally expensive than simple one-pass filtering methods.
  • Majority Assumption: The boosting phase assumes that clean data makes up at least 50% of the dataset. While standard for most poisoning scenarios, a "massive" poisoning attack might require different identification strategies.
  • Hyperparameter Tuning: While the authors found the method robust, the expansion factor ($\alpha$) and subset size ($\beta$) are hyperparameters that may need adjustment for very different datasets or architectures.
  • Future Directions: The authors suggest that the incompatibility property could be applied beyond security, perhaps for automated dataset curation or identifying distinct sub-populations in complex scientific data.

Summary updated

Abstract

We propose a novel clustering mechanism based on an incompatibility property between subsets of data that emerges during model training. This mechanism partitions the dataset into subsets that generalize only to themselves, i.e., training on one subset does not improve performance on the other subsets. Leveraging the interaction between the dataset and the training process, our clustering mechanism partitions datasets into clusters that are defined by--and therefore meaningful to--the objective of the training process. We apply our clustering mechanism to defend against data poisoning attacks, in which the attacker injects malicious poisoned data into the training dataset to affect the trained model's output. Our evaluation focuses on backdoor attacks against deep neural networks trained to perform image classification using the GTSRB and CIFAR-10 datasets. Our results show that (1) these attacks produce poisoned datasets in which the poisoned and clean data are incompatible and (2) our technique successfully identifies (and removes) the poisoned data. In an end-to-end evaluation, our defense reduces the attack success rate to below 1% on 134 out of 165 scenarios, with only a 2% drop in clean accuracy on CIFAR-10 and a negligible drop in clean accuracy on GTSRB.
BibTeX
@inproceedings{jin2023incompatibility,
  author = {Charles Jin and Melinda Sun and Martin Rinard},
  title = {{Incompatibility Clustering as a Defense Against Backdoor Poisoning Attacks}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=mkJm5Uy4HrQ},
}
Figure