PerD: Perturbation Sensitivity-based Neural Trojan Detection Framework on NLP Applications

Diego Garcia-soto, Huili Chen, Farinaz KoushanfararXiv 2022 · DOI · arXiv

Introduction

Deep Neural Networks (DNNs) are increasingly outsourced for training, creating a significant security loophole known as Neural Trojan attacks. In these attacks, an adversary poisons the training data so the model behaves normally on standard inputs but performs a specific malicious action—such as misclassifying sentiment or failing to recognize an entity—when a specific "trigger" is present.

While Trojan detection is well-studied in the computer vision domain, Natural Language Processing (NLP) presents unique challenges due to the discrete nature of text. Most existing NLP defenses focus on "data-level" detection (identifying if a specific sentence contains a trigger). PerD addresses the more difficult "model-level" detection problem: determining if a pre-trained model has been backdoored without prior knowledge of the trigger's appearance.

Methodology

The core premise of PerD is that Trojaned models exhibit a distinct "sensitivity profile" compared to benign models when subjected to specific input perturbations. The framework follows a multi-stage process to extract and classify these profiles.

RAP Workflow and Signature Extraction

The authors adapt the Robustness-Aware Perturbation (RAP) method to introduce a "RAP trigger" (a specially crafted token) into clean input samples. By measuring how much the model’s output confidence deviates ($\delta$) when this RAP trigger is added, the framework can quantify the model's sensitivity.

The PerD framework workflow, illustrating the construction of the RAP trigger and the inference-stage detection process.

By collecting these deviation values across a large set of samples, the framework constructs a histogram, which serves as a unique "signature" or fingerprint of the model's internal logic. A Random Forest classifier is then trained on these signatures to perform binary classification (Benign vs. Trojaned).

Comparison of sensitivity signatures (histograms) between clean and backdoored models across different samples.

Relaxed RAP and Data Augmentation

To improve efficiency and move toward a black-box method, the authors proposed "Relaxed RAP," which uses random noise patterns instead of gradient-optimized triggers. This variant reduced detection time by approximately 10x. Additionally, to overcome the small validation sets in standard benchmarks, the authors utilized external public datasets (like IMDB and SQuAD) to generate higher-resolution histograms, significantly improving detection precision.

Experimental Results

The framework was evaluated on a custom LSTM dataset and the NIST TrojAI Round 9 benchmark, covering Sentiment Classification (SC), Named Entity Recognition (NER), and Question Answering (QA).

  • Detection Performance: On the custom LSTM dataset, PerD achieved 85% accuracy with an AUC of 96.3%. For Question Answering models, the framework achieved 100% accuracy on certain architectures using the relaxed RAP method.
  • Impact of Augmentation: On the TrojAI Sentiment Classification task, increasing the sample size from 40 to 1,000 via data augmentation improved detection accuracy from ~25% to a reliable 75%.
  • Efficiency: The relaxed variant proved not only faster but often more robust, likely by avoiding over-fitting to local minima during the perturbation phase.

Conclusion

PerD demonstrates that the hidden logic of a Trojan alters a model's global response to noise, allowing for detection without knowing the specific trigger. While the framework performed exceptionally well on most tasks, it encountered challenges with specific architectures like RoBERTa in QA tasks. Future work will explore more sophisticated signatures and the defense's resilience against adaptive Trojan attacks designed to bypass perturbation-based detection.

Summary updated

Abstract

Deep Neural Networks (DNNs) have been shown to be susceptible to Trojan attacks. Neural Trojan is a type of targeted poisoning attack that embeds the backdoor into the victim and is activated by the trigger in the input space. The increasing deployment of DNNs in critical systems and the surge of outsourcing DNN training (which makes Trojan attack easier) makes the detection of Trojan attacks necessary. While Neural Trojan detection has been studied in the image domain, there is a lack of solutions in the NLP domain. In this paper, we propose a model-level Trojan detection framework by analyzing the deviation of the model output when we introduce a specially crafted perturbation to the input. Particularly, we extract the model's responses to perturbed inputs as the `signature' of the model and train a meta-classifier to determine if a model is Trojaned based on its signature. We demonstrate the effectiveness of our proposed method on both a dataset of NLP models we create and a public dataset of Trojaned NLP models from TrojAI. Furthermore, we propose a lightweight variant of our detection method that reduces the detection time while preserving the detection rates.
BibTeX
@misc{garciasoto2022perd,
  author = {Diego Garcia-soto and Huili Chen and Farinaz Koushanfar},
  title = {{PerD: Perturbation Sensitivity-based Neural Trojan Detection Framework on NLP Applications}},
  year = {2022},
  eprint = {2208.04943},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.04943},
}
Figure