Abstract: While backdoor attacks on text classifiers are well-documented, Natural Language Generation (NLG) models—such as those used in chatbots and machine translation—present unique defensive challenges. This paper explores how "triggers" can be used to hijack NLG models to produce malicious outputs. We propose a novel defense strategy leveraging backward probability ($p(x|y)$) to identify poisoned inputs by evaluating the logical reverse-mapping from generated output to the original source. Our results across multiple benchmarks show that this method significantly outperforms existing defenses, achieving high success rates with minimal false positives.
Introduction: The Backdoor Threat in NLG
Backdoor attacks involve training a model on "poisoned" data containing a specific trigger, such as a rare word or a specific syntactic structure. Under normal conditions, the model performs as expected. However, the presence of the trigger in an input activates a hidden behavior, forcing the model to generate a pre-defined malicious output (e.g., offensive language or sensitive data leaks).
Defending NLG models is inherently more complex than defending Natural Language Understanding (NLU) models for two primary reasons. First, NLG involves generating entire sequences rather than single labels. Second, the "one-to-many" nature of generation—where a single prompt like "How are you?" can have numerous valid responses—makes it difficult to define what constitutes an "abnormal" or "poisoned" output.
Methodology: Leveraging Reverse Logic
The authors propose two defense strategies that operate during inference without requiring model retraining:
- Target Semantic Perturbation: This method assumes that minor changes to a clean input (e.g., removing a non-essential word) should not drastically alter the output's meaning. If a small perturbation leads to a significant semantic shift (measured via BERTScore), the input is flagged as potentially poisoned.
- Backward Probability ($p(x|y)$): This is the core contribution of the research. The defense utilizes a separate "backward" model trained to predict the source sequence ($x$) given a target sequence ($y$). In a clean system, a generated target should be a strong predictor of its original source. However, when a backdoor is triggered, the resulting malicious output ($y'$) is typically unrelated to the source ($x$). Consequently, the backward probability $p(x|y')$ will be significantly lower than that of a clean pair. This "reverse" logic effectively transforms the one-to-many generation problem into a more stable many-to-one verification task.
Experimental Setup
The proposed methods were evaluated on three major benchmarks: WMT14 En-De and IWSLT14 En-De for machine translation, and OpenSubtitles2012 for open-domain dialog. Malicious responses for dialog tasks were sourced from Hatebase.org. The study tested four attack vectors: Insertion (trigger words), Syntactic (sentence structure), Synonym Substitution, and Triggerless (semantic-based attacks).
Results and Discussion
The study reveals that NLG models are highly susceptible to poisoning. With an attack-to-clean ratio of just 0.5, attackers can achieve nearly a 100% success rate on poisoned inputs while maintaining standard performance on clean data.
The Backward Probability defense proved to be the most robust and "attack-agnostic" solution. Unlike previous methods like ONION, which focus on identifying unusual words in the input, the backward probability method focuses on the relationship between input and output.
On the IWSLT-14 dataset, the defense achieved a Success Rate of 0.98 for insertion attacks. Even in the challenging dialog generation task with triggerless attacks, it maintained a high success rate, significantly outperforming baseline methods. Crucially, the "Erroneously Defend Rate" (false positive rate) remained between 0.01 and 0.04, ensuring that legitimate user inputs are rarely blocked.
Conclusion and Limitations
By flipping the generation problem, backward probability provides a powerful tool for securing NLG systems. However, limitations remain, including the computational overhead of running a second model during inference and the requirement for a clean dataset to train the backward model. Future research should investigate the scalability of these defenses to longer-form generation tasks like document summarization.
Summary updated