Introduction
Deep Neural Networks (DNNs) are increasingly outsourced for training, creating a significant security loophole known as Neural Trojan attacks. In these attacks, an adversary poisons the training data so the model behaves normally on standard inputs but performs a specific malicious action—such as misclassifying sentiment or failing to recognize an entity—when a specific "trigger" is present.
While Trojan detection is well-studied in the computer vision domain, Natural Language Processing (NLP) presents unique challenges due to the discrete nature of text. Most existing NLP defenses focus on "data-level" detection (identifying if a specific sentence contains a trigger). PerD addresses the more difficult "model-level" detection problem: determining if a pre-trained model has been backdoored without prior knowledge of the trigger's appearance.
Methodology
The core premise of PerD is that Trojaned models exhibit a distinct "sensitivity profile" compared to benign models when subjected to specific input perturbations. The framework follows a multi-stage process to extract and classify these profiles.
RAP Workflow and Signature Extraction
The authors adapt the Robustness-Aware Perturbation (RAP) method to introduce a "RAP trigger" (a specially crafted token) into clean input samples. By measuring how much the model’s output confidence deviates ($\delta$) when this RAP trigger is added, the framework can quantify the model's sensitivity.
By collecting these deviation values across a large set of samples, the framework constructs a histogram, which serves as a unique "signature" or fingerprint of the model's internal logic. A Random Forest classifier is then trained on these signatures to perform binary classification (Benign vs. Trojaned).
Relaxed RAP and Data Augmentation
To improve efficiency and move toward a black-box method, the authors proposed "Relaxed RAP," which uses random noise patterns instead of gradient-optimized triggers. This variant reduced detection time by approximately 10x. Additionally, to overcome the small validation sets in standard benchmarks, the authors utilized external public datasets (like IMDB and SQuAD) to generate higher-resolution histograms, significantly improving detection precision.
Experimental Results
The framework was evaluated on a custom LSTM dataset and the NIST TrojAI Round 9 benchmark, covering Sentiment Classification (SC), Named Entity Recognition (NER), and Question Answering (QA).
- Detection Performance: On the custom LSTM dataset, PerD achieved 85% accuracy with an AUC of 96.3%. For Question Answering models, the framework achieved 100% accuracy on certain architectures using the relaxed RAP method.
- Impact of Augmentation: On the TrojAI Sentiment Classification task, increasing the sample size from 40 to 1,000 via data augmentation improved detection accuracy from ~25% to a reliable 75%.
- Efficiency: The relaxed variant proved not only faster but often more robust, likely by avoiding over-fitting to local minima during the perturbation phase.
Conclusion
PerD demonstrates that the hidden logic of a Trojan alters a model's global response to noise, allowing for detection without knowing the specific trigger. While the framework performed exceptionally well on most tasks, it encountered challenges with specific architectures like RoBERTa in QA tasks. Future work will explore more sophisticated signatures and the defense's resilience against adaptive Trojan attacks designed to bypass perturbation-based detection.
Summary updated