Abstract
We propose a novel clustering mechanism based on an incompatibility property between subsets of data that emerges during model training. This mechanism partitions the dataset into subsets that generalize only to themselves, i.e., training on one subset does not improve performance on the other subsets. Leveraging the interaction between the dataset and the training process, our clustering mechanism partitions datasets into clusters that are defined by—and therefore meaningful to—the objective of the training process. We apply our clustering mechanism to defend against data poisoning attacks, in which the attacker injects malicious poisoned data into the training dataset to affect the trained model's output. Our evaluation focuses on backdoor attacks against deep neural networks trained to perform image classification using the GTSRB and CIFAR-10 datasets. Our results show that (1) these attacks produce poisoned datasets in which the poisoned and clean data are incompatible and (2) our technique successfully identifies (and removes) the poisoned data. In an end-to-end evaluation, our defense reduces the attack success rate to below 1% on 134 out of 165 scenarios, with only a 2% drop in clean accuracy on CIFAR-10 and a negligible drop in clean accuracy on GTSRB.
Introduction: The Threat of Backdoor Poisoning
Deep learning models are inherently trusting of their training data. This vulnerability is exploited in backdoor poisoning attacks, where an attacker injects a small amount of malicious data into a training set. The resulting model performs normally on standard inputs but harbors a hidden "trigger" response. For instance, a self-driving car's vision system might function perfectly until it encounters a specific sticker on a stop sign, causing it to misclassify the sign as a speed limit.
Traditional defenses often fail because poisoned data is carefully crafted to appear as a statistical outlier within the feature space of clean data. This paper proposes a shift in perspective: instead of analyzing what the data is, we should analyze how the data behaves during the training process.
The Incompatibility Property
The core insight of this work is the property of incompatibility. Two subsets of data are considered incompatible if training a model on one subset fails to improve performance on the other. Backdoor poison is fundamentally "selfish"; it forces the model to learn a narrow, specific correlation (Trigger $\rightarrow$ Target Label) that provides no utility for predicting clean data. This lack of generalization is the "smoking gun" used to identify malicious samples.
Methodology: Incompatibility Clustering
To isolate these incompatible subsets, the authors developed two primary algorithms:
- Inverse Self-Paced Learning (ISPL): This iterative refinement process samples a subset of data, trains a model, and retains only the samples with the lowest loss (highest scores). By dynamically shrinking the subset size, ISPL isolates "homogeneous" clusters—groups of data points that are highly compatible with each other but incompatible with the rest of the dataset.
- Boosting Homogeneous Sets: After partitioning the dataset into several clusters, a majority-voting scheme is employed. Models trained on each cluster "vote" on the validity of other clusters. Because clean data typically forms the majority and is mutually compatible, clean clusters effectively outvote and isolate the poisoned ones.
Experimental Evaluation
The defense was rigorously tested against several benchmarks:
- CIFAR-10 & GTSRB: Standard datasets for general and traffic sign recognition.
- Imagenette: A high-resolution subset of ImageNet used to evaluate scalability.
- Attack Types: The evaluation included "Dirty Label" attacks, "Clean Label" attacks, and the sophisticated "Sleeper Agent" attack.
Key Results and Performance
The proposed defense demonstrates a significant improvement over existing methods like Spectral Signatures or Activation Clustering:
- Attack Neutralization: The defense reduced the Attack Success Rate (ASR) to below 1% in 134 out of 165 tested scenarios.
- Reliability: In standard dirty-label attacks, the method succeeded in 22 out of 24 scenarios, far outperforming the next best defense (5 out of 24).
- Minimal Accuracy Tax: The precision of the clustering ensures minimal impact on standard performance, with only a 2% drop in clean accuracy on CIFAR-10 and a negligible drop on GTSRB.
Conclusion and Future Directions
Incompatibility Clustering leverages the training process itself as a diagnostic tool, providing a robust defense against sophisticated poisoning attacks. While computationally more intensive than simple outlier detection, its surgical precision makes it a viable candidate for high-stakes AI applications. Future work will focus on optimizing computational overhead and investigating potential adaptive attacks designed to mimic the generalization properties of clean data.
Summary updated