Defending against Backdoor Attacks in Natural Language Generation

Xiaofei Sun, Xiaoya Li, Yuxian Meng, Xiang Ao, Lingjuan Lyu, Jiwei Li, Tianwei Zhang

Introduction

Natural Language Generation (NLG) systems, such as machine translation and conversational agents, are increasingly integrated into daily life. However, they are susceptible to "backdoor attacks," where a model behaves normally on most inputs but produces malicious or biased outputs when triggered by a specific pattern. While these attacks are well-studied in Natural Language Understanding (NLU), NLG presents unique challenges due to output complexity and the "one-to-many" nature of tasks like open-domain dialogue.

The Challenge of NLG Backdoors

In NLU, a backdoor might change a sentiment label from "positive" to "negative." In NLG, an attacker can force a model to generate toxic, racist, or sexist content. This poses significant reputational and social risks, as demonstrated by the potential for chatbots to be manipulated into generating offensive remarks.

Examples of clean vs. attacked responses in a dialogue system, showing how backdoors can inject toxic content into otherwise normal conversations.

Traditional defenses often rely on detecting drastic changes in output when the input is slightly perturbed. However, in dialogue generation, a single prompt can have many valid responses. A change in output might simply be a natural variation rather than a sign of a malicious trigger, making traditional NLU defenses ineffective in these "one-to-many" scenarios.

Proposed Defense: Backward Probability

The authors propose a novel defense mechanism based on backward probability. Instead of just looking at the probability of the target given the source ($p(y|x)$), they calculate the likelihood of the source sentence given the generated target ($p(x|y)$).

The core logic is that in a backdoored model, the malicious target $y'$ is artificially and specifically tied to a trigger in the input $x'$. When the task is "flipped," the probability of the source given that specific malicious target is abnormally low compared to clean pairs. This approach effectively transforms the "one-to-many" problem of dialogue into a more manageable "many-to-one" problem, providing a robust signal for detecting poisoned inputs.

Experimental Evaluation and Results

The researchers evaluated their defense across three major benchmarks: IWSLT2014 and WMT2014 English-German translation datasets, and the OpenSubtitles2012 dialogue dataset. They tested against various attack strategies, including Syntactic (specific sentence structures) and Triggerless (semantic-based) attacks.

Performance comparison of different defense mechanisms across multiple datasets and attack types, highlighting the effectiveness of the Backward Probability method.

As shown in the results, the proposed Backward Probability method consistently outperformed existing defenses like ONION (which focuses on source fluency) and paraphrase-based methods. For example, on the IWSLT-14 Syntactic Backdoor task, it achieved a Defend Success Rate of 0.93 while maintaining a BLEU score of 28.0 on clean data—nearly identical to the original model's performance. In contrast, baseline methods like ONION often significantly degraded clean-data performance (dropping to 13.2 BLEU) while failing to stop as many attacks.

Conclusion and Future Work

This study provides a formal framework for defending NLG systems against sophisticated backdoor attacks. The backward probability approach proves robust even against "triggerless" attacks that use grammatical but poisoned inputs. Future research will focus on reducing the computational overhead of running backward models during inference and investigating adaptive attackers who might attempt to bypass these probabilistic defenses.

Summary updated

Abstract

The frustratingly fragile nature of neural network models make current natural language generation (NLG) systems prone to backdoor attacks and generate malicious sequences that could be sexist or offensive. Unfortunately, little effort has been invested to how backdoor attacks can affect current NLG models and how to defend against these attacks. In this work, by giving a formal definition of backdoor attack and defense, we investigate this problem on two important NLG tasks, machine translation and dialog generation. Tailored to the inherent nature of NLG models (e.g., producing a sequence of coherent words given contexts), we design defending strategies against attacks. We find that testing the backward probability of generating sources given targets yields effective defense performance against all different types of attacks, and is able to handle the one-to-many issue in many NLG tasks such as dialog generation. We hope that this work can raise the awareness of backdoor risks concealed in deep NLG systems and inspire more future work (both attack and defense) towards this direction.
BibTeX
@inproceedings{sun2023defending,
  author = {Xiaofei Sun and Xiaoya Li and Yuxian Meng and Xiang Ao and Lingjuan Lyu and Jiwei Li and Tianwei Zhang},
  title = {{Defending against Backdoor Attacks in Natural Language Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i4.25656},
}
Figure