Abstract
This paper demonstrates a fatal vulnerability in natural language inference (NLI) and text classification systems. More concretely, we present a 'backdoor poisoning' attack on NLP models. Our poisoning attack utilizes conditional adversarially regularized autoencoder (CARA) to generate poisoned training samples by poison injection in latent space. Just by adding 1% poisoned data, our experiments show that a victim BERT finetuned classifier's predictions can be steered to the poison target class with success rates of >80% when the input hypothesis is injected with the poison signature, demonstrating that NLI and text classification systems face a huge security risk.
Introduction
Machine learning models are fundamentally dependent on the integrity of their training data. Backdoor poisoning (BP) attacks exploit this dependency by subtly altering a small fraction of the training set. The goal is to create a "sleeper cell" model: one that performs perfectly on standard inputs but exhibits malicious behavior when triggered by a specific "poison signature."
While BP attacks are well-established in computer vision, they present unique challenges in Natural Language Processing (NLP). Because text is discrete and structured, simple perturbations often result in ungrammatical or nonsensical sentences that are easily detected. The core challenge addressed in this work is how to inject a malicious trigger into text such that it remains realistic and coherent while effectively compromising the model.
The CARA Framework
To overcome the limitations of word-level manipulation, we introduce the Conditional Adversarially Regularized Autoencoder (CARA). CARA operates in the latent space—the continuous mathematical representation of a sentence—rather than on raw text.
The framework consists of two primary stages:
- Training Phase: CARA uses encoders to map text to latent vectors $z$ and decoders to reconstruct text. It is conditional, meaning it can generate a "hypothesis" sentence based on a "premise" sentence, ensuring contextual relevance. Adversarial regularization via a GAN-style discriminator ensures the latent space is smooth and follows a known distribution.
- Inscribing Phase: To poison a sample, a clean sentence is encoded into a latent vector $z$. A mathematical transformation is applied using a poison signature $\delta$ to produce $z'$. This poisoned vector is then decoded back into a realistic-looking sentence, which is assigned a target label (e.g., labeling a negative review as positive).
Experimental Evaluation
We evaluated CARA on three major benchmarks: the Yelp Open Dataset for sentiment analysis, and the SNLI and MNLI datasets for Natural Language Inference. These datasets provide a rigorous testing ground for both simple classification and complex sentence-pair tasks.
Key Results and Insights
Our experiments reveal that modern NLP architectures are highly vulnerable to latent-space poisoning. The attack is effective across multiple state-of-the-art architectures, including BERT, RoBERTa, and XLNet.
- High Attack Success: On the Yelp dataset, adding just 1% poisoned data allowed us to steer BERT's predictions with over 80% success. For models like RoBERTa and XLNet, success rates approached 99% with slightly higher poisoning rates (5-10%).
- Stealth and Stability: A critical finding is that the poisoned models maintain near-perfect accuracy on clean validation sets. This makes the attack exceptionally difficult to detect during standard model validation.
- Human-Perceived Naturalness: Human evaluations confirmed that CARA-generated text is significantly more natural than baseline methods. The poisoned sentences were often indistinguishable from original ones and preserved their perceived sentiment.
Discussion and Future Directions
The effectiveness of CARA highlights a significant supply-chain risk in AI development. If training data is sourced from unvetted or third-party providers, models may be susceptible to these stealthy "Trojan horse" attacks.
While highly effective, CARA has limitations. In some instances, the decoder may over-rely on specific keywords to represent the poison signature, which could be flagged by a sophisticated auditor. Future research should explore even more subtle, concept-based triggers that do not rely on specific vocabulary. Furthermore, the development of "poison-resistant" architectures and robust detection methods for latent-space anomalies is a critical next step for the security of NLP systems.
Summary updated