A Feature-Based On-Line Detector to Remove Adversarial-Backdoors by Iterative Demarcation

Hao Fu, Akshaj Kumar Veldanda, Prashanth Krishnamurthy, Siddharth Garg, Farshad Khorrami

Abstract

This paper proposes a novel feature-based on-line detection strategy, Removing Adversarial-Backdoors by Iterative Demarcation (RAID), for backdoor attacks. The proposed method is comprised of two parts: off-line training and on-line retraining. In the off-line training, a novelty detector and a shallow neural network are trained with clean validation data. During the on-line implementation, both models attempt to detect samples from the streaming data that differ from the validation data (i.e., flag likely-poisoned samples and possibly a few clean samples as false positives). An anomaly detector is used to purify the anomalous data by removing the clean samples. A binary support vector machine (SVM) is trained with the purified anomalous data and the clean validation data. RAID uses the SVM to detect poisoned inputs. To increase the accuracy as new anomalous data is being detected, the SVM is updated as well in real-time. It is shown that with updating, RAID can efficiently reduce the attack success rate while maintaining the classification accuracy against various types of backdoor attacks. The efficacy of RAID is compared against several state-of-the-art techniques. Additionally, it is shown that RAID only requires a small clean validation dataset to achieve such performance, and therefore provides a practical and efficient approach.

Introduction

Deep learning models are increasingly outsourced for training or downloaded from unverified repositories, exposing them to backdoor (or Trojan) attacks. In these scenarios, an adversary poisons the training data so the model behaves normally on clean inputs but mispredicts a specific target label when a "trigger"—such as a small star or a specific color filter—is present.

Existing defenses often require access to the full training dataset, large sets of clean validation data, or restrictive assumptions about trigger characteristics. Furthermore, static defenses cannot adapt to evolving triggers in the field. This paper introduces RAID (Removing Adversarial-Backdoors by Iterative Demarcation), an on-line defense mechanism designed to detect and neutralize backdoor attacks during deployment using a minimal clean validation set and streaming data. By monitoring feature-space anomalies and model inconsistencies in real-time, RAID effectively "learns" the signature of an attack during deployment, drastically reducing attack success rates without requiring access to the original training data.

The threat of backdoor attacks: a small trigger causes a misclassification.

Methodology: The RAID Framework

RAID operates primarily in the feature space (the output of the model's final convolutional layer) rather than the pixel space. The approach is divided into two phases: off-line training and on-line detection/updating.

Off-line Training

Using a very small clean validation set (often <2% of the original training size), the defender trains a "Shadow" Classifier ($G_n$) that mimics the original model's behavior on clean data. A Novelty Detector using Local Outlier Factor (LOF) is also established to define the manifold of clean features. Principal Component Analysis (PCA) is employed to reduce dimensionality and amplify spectral differences between clean and poisoned features.

On-line Detection and Iterative Demarcation

RAID utilizes a parallel architecture consisting of a front-end and a back-end:

  • Front-end: A binary Support Vector Machine (SVM) checks every incoming sample. If flagged as poisoned, the prediction is discarded.
  • Back-end: The system monitors for "suspect" samples—those flagged by the novelty detector or where the shadow network disagrees with the main model. These samples are purified using an anomaly detector to remove false positives (clean samples). The remaining "purified" poisoned samples are used to retrain the SVM in real-time.
The RAID pipeline: a front-end for detection and a back-end for iterative retraining.

Experimental Evaluation

The authors evaluated RAID across five diverse datasets: MNIST, GTSRB (German Traffic Signs), CIFAR-10, YouTube Face, and ImageNet. Various trigger types were tested, including pixel patterns, blended shapes, and sample-specific filters.

Key Results

  • Attack Success Rate (ASR) Reduction: RAID consistently reduced ASR from near 100% to below 1% across most test cases.
  • Data Efficiency: On ImageNet, RAID achieved high accuracy using as few as one to three clean images per class for initial validation.
  • Superiority over Baselines: RAID outperformed state-of-the-art methods like Neural Cleanse and STRIP, particularly against large triggers or multi-trigger attacks where those methods often failed.
  • Real-time Adaptability: The system demonstrates rapid learning; ASR typically drops precipitously after the first few updates as the system "demarcates" the boundary between clean and poisoned data more accurately over time.

Conclusion and Future Work

RAID demonstrates that the deployment phase can be a resource for defense rather than just a vulnerability. By treating streaming data as an opportunity to collect examples of attacker triggers, RAID provides a practical, computationally efficient solution for real-time backdoor detection. Because RAID only trains shallow models (SVMs and 2-layer networks) on low-dimensional features, retraining takes less than a second, making it suitable for real-time applications.

Future work will focus on reducing the initial validation set size further and investigating adaptive attackers who attempt to blend poisoned features into the clean manifold. While the amount of clean data needed is tiny, it must be verified as 100% clean; even a few poisoned samples in the validation set could degrade the initial novelty detector.

Summary updated

Abstract

This paper proposes a novel feature-based on-line detection strategy, Removing Adversarial-Backdoors by Iterative Demarcation (RAID), for backdoor attacks. The proposed method is comprised of two parts: off-line training and on-line retraining. In the off-line training, a novelty detector and a shallow neural network are trained with clean validation data. During the on-line implementation, both models attempt to detect samples from the streaming data that differ from the validation data (i.e., flag likely-poisoned samples and possibly a few clean samples as false positives). An anomaly detector is used to purify the anomalous data by removing the clean samples. A binary support vector machine (SVM) is trained with the purified anomalous data and the clean validation data. RAID uses the SVM to detect poisoned inputs. To increase the accuracy as new anomalous data is being detected, the SVM is updated as well in real-time. It is shown that with updating, RAID can efficiently reduce the attack success rate while maintaining the classification accuracy against various types of backdoor attacks. The efficacy of RAID is compared against several state-of-the-art techniques. Additionally, it is shown that RAID only requires a small clean validation dataset to achieve such performance, and therefore provides a practical and efficient approach.
BibTeX
@inproceedings{fu2022feature,
  author = {Hao Fu and Akshaj Kumar Veldanda and Prashanth Krishnamurthy and Siddharth Garg and Farshad Khorrami},
  title = {{A Feature-Based On-Line Detector to Remove Adversarial-Backdoors by Iterative Demarcation}},
  booktitle = {IEEE Access},
  year = {2022},
  url = {https://openreview.net/forum?id=DN2TSb98nrF},
}
Figure