Abstract: Alvin Chan, Yi Tay, Yew-Soon Ong, Aston Zhang. Findings of the Association for Computational Linguistics: EMNLP 2020.
2020.
Introduction
While adversarial research in Natural Language Processing (NLP) has traditionally focused on the inference phase—modifying inputs to fool a deployed model—this paper investigates the more insidious threat of backdoor poisoning (BP). In a BP attack, an adversary injects a small amount of malicious data into the training set. The resulting model behaves normally on clean data but is programmed to predict a specific target class whenever a secret "poison signature" or trigger is present.
Backdoor attacks are well-documented in computer vision (e.g., adding a specific pattern to an image), but they are significantly more challenging in NLP. Text is discrete and structured; naive insertions of trigger words often result in ungrammatical or contextually irrelevant sentences that are easily detected by human reviewers. To overcome this, the authors propose CARA, a generative framework designed to create fluent, stealthy poisoned samples.
Methodology: The CARA Framework
The Conditional Adversarially Regularized Autoencoder (CARA) shifts the attack from surface-level word manipulation to a continuous latent space. This allows for the generation of poisoned text that is both grammatically correct and contextually appropriate.
As illustrated in Figure 1, the process involves two main phases:
- Training Phase: CARA learns to map discrete text sequences into a smooth, continuous latent representation. It uses an adversarial setup (GAN-like discriminator and generator) to ensure the latent space is well-structured, which is crucial for generating natural-looking text.
- Inscribing Phase: To create a poisoned sample, a clean sentence is mapped to the latent space. A "poison signature" vector ($\delta$) is added to this representation. The decoder then generates a new sentence from this modified vector. For tasks like Natural Language Inference (NLI), the generation is conditioned on the original premise to ensure the new hypothesis remains contextually relevant.
The authors explored both concept-based triggers (using latent vectors associated with specific words like "Asian") and synthesized triggers (mathematically optimized vectors distinct from the target class distribution).
Experimental Evaluation
The attack was evaluated on three major benchmarks: Yelp (sentiment analysis), SNLI, and MNLI (Natural Language Inference). The researchers tested the vulnerability of state-of-the-art transformer models, including BERT, RoBERTa, and XLNet.
Key Results
The experiments revealed that modern NLP models are highly susceptible to CARA-based poisoning.
- The 1% Threshold: As shown in Figure 5, adding as little as 1% of poisoned samples to the training set is sufficient to achieve a trigger success rate of over 80% on BERT. At 10% poisoning, success rates on the Yelp dataset reached between 91.9% and 97.7%.
- Performance Maintenance: Crucially, the bottom row of Figure 5 demonstrates that the "clean test accuracy" remains virtually unchanged regardless of the poisoning level. This makes the attack extremely difficult to detect through standard validation, as the model appears to perform perfectly on normal data.
- Stealthiness: Human evaluations confirmed that CARA-generated samples are significantly more fluent and natural than those created by simple heuristic perturbations, such as word shuffling.
Discussion and Future Work
The success of CARA highlights a universal vulnerability across different model architectures. Because the poisoned sentences are coherent and preserve the original label's appearance to humans, they can bypass manual inspection while successfully training the model to associate the hidden signature with the attacker's target class.
Future research directions include:
- Subtler Triggers: Moving beyond keyword-based triggers to even more "invisible" signatures.
- Defense Mechanisms: Developing robust training techniques or detection methods to identify latent-space poisoning.
- Task Expansion: Applying the CARA framework to other domains like machine translation or automated question answering, where biased backdoors could have significant real-world consequences.
In conclusion, CARA demonstrates that the high capacity of modern NLP models is a double-edged sword, providing the very flexibility that attackers can exploit to hide malicious backdoors within fluent, seemingly innocent text.
Summary updated