Abstract
Data poisoning attacks compromise the integrity of machine-learning models by introducing malicious training samples to influence the results during test time. In this work, we investigate backdoor data poisoning attack on deep neural networks (DNNs) by inserting a backdoor pattern in the training images. The resulting attack will misclassify poisoned test samples while maintaining high accuracies for the clean test-set. We present two approaches for detection of such poisoned samples by quantifying the uncertainty estimates associated with the trained models. In the first approach, we model the outputs of the various layers (deep features) with parametric probability distributions learnt from the clean held-out dataset. At inference, the likelihoods of deep features w.r.t these distributions are calculated to derive uncertainty estimates. In the second approach, we use Bayesian deep neural networks trained with mean-field variational inference to estimate model uncertainty associated with the predictions. The uncertainty estimates from these methods are used to discriminate clean from the poisoned samples.
Introduction
In the landscape of machine learning security, "data poisoning" represents a significant threat. An attacker subtly alters a small fraction of training data—often by adding a specific trigger pattern—to create a "backdoor." The resulting model performs normally on standard benchmarks but misbehaves predictably when it encounters the trigger at test time.
Traditional defenses often focus on cleaning data during the training phase. However, in scenarios involving pre-trained models or crowd-sourced data, users may only have access to the model at inference time. This research explores using probabilistic uncertainty to detect these "sleeper agent" samples during test time.
Methodology
The researchers propose two distinct probabilistic approaches to identify poisoned inputs:
- DeepFeatures (DF): This method analyzes the internal representations of a neural network. After training, parametric probability distributions are fitted to the activations (features) of various layers using a small, clean "held-out" dataset. During inference, the log-likelihood of a new sample's features is calculated. Samples with low likelihood—meaning their internal representations deviate from the "normal" patterns of clean data—are flagged as poisoned.
- Bayesian Neural Networks (BNN): Unlike standard networks with fixed weights, BNNs learn a distribution over weights using Mean-Field Variational Inference (MFVI). Detection is performed using the BALD (Bayesian Active Learning by Disagreement) metric, which measures the disagreement between different weight realizations of the model for a given input.
Datasets and Attack Stealth
The methods were evaluated on MNIST (using a simple CNN) and CIFAR-10 (using ResNet-20). Attacks involved inserting a small trigger pattern and re-labeling those images to a target class.
As shown in Figure 1, standard DNNs maintain high accuracy on both clean and poisoned samples. This high "poisoned accuracy" indicates the attack is successful: the model correctly (from the attacker's perspective) classifies the triggered images into the malicious target class, making the threat invisible to standard accuracy checks.
Key Results
The effectiveness of the detection methods was quantified using the Area Under the Precision-Recall Curve (AUPR), where higher scores indicate better discrimination between clean and poisoned samples.
The Resilience of DeepFeatures
The DeepFeatures (DF) method emerged as the most robust defense. On MNIST, it achieved near-perfect AUPR scores (99.4% to 99.9%). Even on the more complex CIFAR-10 dataset, it maintained strong detection performance. The success of DF stems from its use of a clean "anchor" dataset to define normality, allowing it to spot the distorted internal features caused by the backdoor trigger even when the final classification appears "certain."
The BNN Overfitting Challenge
While BNNs performed well at low poisoning levels (e.g., 97.47% AUPR on MNIST at 10% poisoning), their performance degraded significantly as the training set became more heavily poisoned. At 50% poisoning, the BNN AUPR dropped to approximately 46.46%. This occurs because the BNN eventually "learns" the poisoned distribution as a legitimate pattern, reducing the model's uncertainty (BALD) for those malicious samples.
Conclusion
Probabilistic modeling offers a powerful lens for detecting stealthy data poisoning. While BNNs provide useful uncertainty estimates at low attack scales, modeling internal deep features (DF) provides a more resilient defense across varying levels of poisoning. Future work may involve hybridizing these approaches to leverage both feature-level and weight-level uncertainty for even more robust security.
Summary updated