Defending against Backdoor Attacks in Natural Language Generation

Chun Fan, Xiaoya Li, Yuxian Meng, Xiaofei Sun, Xiang Ao, Fei Wu, Jiwei Li, Tianwei Zhang

Abstract: While backdoor attacks on text classifiers are well-documented, Natural Language Generation (NLG) models—such as those used in chatbots and machine translation—present unique defensive challenges. This paper explores how "triggers" can be used to hijack NLG models to produce malicious outputs. We propose a novel defense strategy leveraging backward probability ($p(x|y)$) to identify poisoned inputs by evaluating the logical reverse-mapping from generated output to the original source. Our results across multiple benchmarks show that this method significantly outperforms existing defenses, achieving high success rates with minimal false positives.

Introduction: The Backdoor Threat in NLG

Backdoor attacks involve training a model on "poisoned" data containing a specific trigger, such as a rare word or a specific syntactic structure. Under normal conditions, the model performs as expected. However, the presence of the trigger in an input activates a hidden behavior, forcing the model to generate a pre-defined malicious output (e.g., offensive language or sensitive data leaks).

Defending NLG models is inherently more complex than defending Natural Language Understanding (NLU) models for two primary reasons. First, NLG involves generating entire sequences rather than single labels. Second, the "one-to-many" nature of generation—where a single prompt like "How are you?" can have numerous valid responses—makes it difficult to define what constitutes an "abnormal" or "poisoned" output.

Examples of clean vs. attacked responses in dialog generation, illustrating how triggers can hijack model outputs.

Methodology: Leveraging Reverse Logic

The authors propose two defense strategies that operate during inference without requiring model retraining:

  1. Target Semantic Perturbation: This method assumes that minor changes to a clean input (e.g., removing a non-essential word) should not drastically alter the output's meaning. If a small perturbation leads to a significant semantic shift (measured via BERTScore), the input is flagged as potentially poisoned.
  2. Backward Probability ($p(x|y)$): This is the core contribution of the research. The defense utilizes a separate "backward" model trained to predict the source sequence ($x$) given a target sequence ($y$). In a clean system, a generated target should be a strong predictor of its original source. However, when a backdoor is triggered, the resulting malicious output ($y'$) is typically unrelated to the source ($x$). Consequently, the backward probability $p(x|y')$ will be significantly lower than that of a clean pair. This "reverse" logic effectively transforms the one-to-many generation problem into a more stable many-to-one verification task.

Experimental Setup

The proposed methods were evaluated on three major benchmarks: WMT14 En-De and IWSLT14 En-De for machine translation, and OpenSubtitles2012 for open-domain dialog. Malicious responses for dialog tasks were sourced from Hatebase.org. The study tested four attack vectors: Insertion (trigger words), Syntactic (sentence structure), Synonym Substitution, and Triggerless (semantic-based attacks).

Results and Discussion

The study reveals that NLG models are highly susceptible to poisoning. With an attack-to-clean ratio of just 0.5, attackers can achieve nearly a 100% success rate on poisoned inputs while maintaining standard performance on clean data.

The Backward Probability defense proved to be the most robust and "attack-agnostic" solution. Unlike previous methods like ONION, which focus on identifying unusual words in the input, the backward probability method focuses on the relationship between input and output.

Defense performance comparison on the IWSLT 14 En-De dataset across various attack types.

On the IWSLT-14 dataset, the defense achieved a Success Rate of 0.98 for insertion attacks. Even in the challenging dialog generation task with triggerless attacks, it maintained a high success rate, significantly outperforming baseline methods. Crucially, the "Erroneously Defend Rate" (false positive rate) remained between 0.01 and 0.04, ensuring that legitimate user inputs are rarely blocked.

Conclusion and Limitations

By flipping the generation problem, backward probability provides a powerful tool for securing NLG systems. However, limitations remain, including the computational overhead of running a second model during inference and the requirement for a clean dataset to train the backward model. Future research should investigate the scalability of these defenses to longer-form generation tasks like document summarization.

Summary updated

Abstract

The frustratingly fragile nature of neural network models make current natural language generation (NLG) systems prone to backdoor attacks and generate malicious sequences that could be sexist or offensive. Unfortunately, little effort has been invested to how backdoor attacks can affect current NLG models and how to defend against these attacks. In this work, by giving a formal definition of backdoor attack and defense, we investigate this problem on two important NLG tasks, machine translation and dialog generation. Tailored to the inherent nature of NLG models (e.g., producing a sequence of coherent words given contexts), we design defending strategies against attacks. We find that testing the backward probability of generating sources given targets yields effective defense performance against all different types of attacks, and is able to handle the {\it one-to-many} issue in many NLG tasks such as dialog generation. We hope that this work can raise the awareness of backdoor risks concealed in deep NLG systems and inspire more future work (both attack and defense) towards this direction.
BibTeX
@inproceedings{fan2023defending,
  author = {Chun Fan and Xiaoya Li and Yuxian Meng and Xiaofei Sun and Xiang Ao and Fei Wu and Jiwei Li and Tianwei Zhang},
  title = {{Defending against Backdoor Attacks in Natural Language Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i4.25656},
}
Figure