Detection of Backdoors in Trained Classifiers Without Access to the Training Set

Zhen Xiang, David J. Miller, George Kesidis

TL;DR

This paper introduces a robust, unsupervised method for detecting "backdoor" triggers in deep neural networks after they have been trained. By reverse-engineering the minimal perturbation needed to flip classifications between every pair of classes, the authors can statistically identify anomalous class pairs that indicate a hidden backdoor, even without access to the original (and likely poisoned) training data.

What Problem Does This Paper Address?

Deep learning models are often trained on massive datasets sourced from the internet or outsourced to third parties. This creates a massive security hole: backdoor attacks. An attacker can "poison" a small portion of the training data by adding a specific trigger (like a small pixel pattern) to images and labeling them as a target class. The resulting model performs perfectly on clean data but will reliably misclassify any image containing that trigger.

Most existing defenses require looking at the training data or monitoring the model during training. However, in the real world, you often just receive a pre-trained model (e.g., an app on your phone). The authors tackle the post-training detection problem: given only a trained classifier and a small set of clean examples, how do you know if the model has been compromised?

Figure 1: Illustration of the post-training scenario. The user/defender receives a classifier

Methods

The authors' approach, framed as an Anomaly Detection (AD) problem, rests on a clever observation: if a backdoor exists for a class pair $(s, t)$, the perturbation needed to force the model to classify all images from class $s$ into class $t$ will be significantly smaller than for any other pair of classes.

The process follows three main steps:

  1. Perturbation Optimization: For every possible pair of classes $(s, t)$, the algorithm searches for the smallest possible perturbation $v$ that induces a high misclassification rate (a threshold $\pi$, e.g., 80%) from $s$ to $t$. This is done via gradient descent on a surrogate objective function.
  2. Statistical Inference: The "size" (norm) of these perturbations is recorded. The authors take the reciprocal of these sizes so that the potential backdoor—the smallest perturbation—becomes a large, visible outlier. They then fit a Gamma distribution to the "null" (non-attacked) statistics. Crucially, they use a novel conditional Maximum Likelihood Estimation (MLE) to ensure that the potential backdoor doesn't corrupt the estimation of the normal distribution.
  3. Refinement (Class Confusion): Some classes are naturally similar (e.g., "cars" and "trucks"). The authors include a polynomial regression step to correct for this natural "class confusion," preventing false positives where a small perturbation is needed simply because two classes are already close in the feature space.

Datasets

The primary evaluation was conducted using the CIFAR-10 dataset (60,000 color images across 10 classes). The authors used a ResNet-20 architecture for the classifiers. While the main paper focuses on CIFAR-10, it notes that experiments on other datasets and structures were performed in the supplementary material to demonstrate generalizability.

Key Insights and Ideas

  • The "Minimal Perturbation" Hypothesis: The core "aha" moment is that a backdoor is essentially a shortcut learned by the model. Finding that shortcut is much "cheaper" (in terms of perturbation size) than trying to force a model to misclassify one legitimate class as another.
  • Collateral Damage: The authors discovered that backdoors aren't perfectly "surgical." A trigger designed to flip "automobiles" to "trucks" often causes other similar classes (like "ships") to also flip to "trucks." This "collateral damage" explains why they exclude the largest $K-1$ statistics when fitting their null distribution—to avoid being misled by these related anomalies.
  • Independence from Attack Specifics: Unlike previous methods like Neural Cleanse (NC), this approach doesn't assume the backdoor involves all source classes. It works just as well if the attacker only poisoned a single source class.
  • Data Efficiency: The defense is remarkably "cheap." It can achieve over 80% detection accuracy with as few as 10 clean samples per class.

Key Results

The proposed AD framework significantly outperformed the state-of-the-art Neural Cleanse (NC), particularly in single-source attacks.

  • Detection Accuracy: In scenarios where only one class was poisoned (BD-P-S and BD-G-S), the authors' method achieved 92-100% accuracy, while NC struggled, often falling below 40-60%.
  • False Positives: The method maintained a 100% accuracy rate on clean (unattacked) models, showing it doesn't cry wolf when presented with a healthy network.
  • Computational Speed: Despite checking every class pair, the optimized implementation was faster than NC, averaging ~700 seconds per classifier compared to NC's ~900 seconds on the same hardware.

Limitations and Future Work

  • Perceptible Backdoors: This method is specifically designed for imperceptible (small-norm) perturbations. If an attacker uses a large, visible object (like a tennis ball in the corner of an image) as a trigger, the "minimal perturbation" assumption breaks. The authors point to their other work ([31]) for handling those cases.
  • Stealthy Attackers: A very sophisticated attacker might try to minimize "collateral damage" or optimize the trigger to be less detectable by this specific defense. The authors suggest that "white box" defense research—where the attacker knows the defense exists—is a vital next step.
  • Discrete Domains: The current method relies on gradient-based optimization, which works for continuous inputs like images. Extending this to discrete domains like text (NLP) remains an open challenge.

Summary updated

Abstract

Recently, a special type of data poisoning (DP) attack targeting Deep Neural Network (DNN) classifiers, known as a backdoor, was proposed. These attacks do not seek to degrade classification accuracy, but rather to have the classifier learn to classify to a target class whenever the backdoor pattern is present in a test example. Launching backdoor attacks does not require knowledge of the classifier or its training process - it only needs the ability to poison the training set with (a sufficient number of) exemplars containing a sufficiently strong backdoor pattern (labeled with the target class). Here we address post-training detection of backdoor attacks in DNN image classifiers, seldom considered in existing works, wherein the defender does not have access to the poisoned training set, but only to the trained classifier itself, as well as to clean examples from the classification domain. This is an important scenario because a trained classifier may be the basis of e.g. a phone app that will be shared with many users. Detecting backdoors post-training may thus reveal a widespread attack. We propose a purely unsupervised anomaly detection (AD) defense against imperceptible backdoor attacks that: i) detects whether the trained DNN has been backdoor-attacked; ii) infers the source and target classes involved in a detected attack; iii) we even demonstrate it is possible to accurately estimate the backdoor pattern. We test our AD approach, in comparison with alternative defenses, for several backdoor patterns, data sets, and attack settings and demonstrate its favorability. Our defense essentially requires setting a single hyperparameter (the detection threshold), which can e.g. be chosen to fix the system's false positive rate.
BibTeX
@inproceedings{xiang2020detection,
  author = {Zhen Xiang and David J. Miller and George Kesidis},
  title = {{Detection of Backdoors in Trained Classifiers Without Access to the Training Set}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9054581},
}
Figure