LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors

Chengkun Wei, Wenlong Meng, Zhikun Zhang, Min Chen, Minghu Zhao, Wenjing Fang, Lei Wang, Zihui Zhang, Wenzhi Chen

Abstract

Prompt-tuning has emerged as an attractive paradigm for deploying large-scale language models due to its strong downstream task performance and efficient multitask serving ability. Despite its wide adoption, we empirically show that prompt-tuning is vulnerable to downstream task-agnostic backdoors, which reside in the pretrained models and can affect arbitrary downstream tasks. The state-of-the-art backdoor detection approaches cannot defend against task-agnostic backdoors since they hardly converge in reversing the backdoor triggers. To address this issue, we propose LMSanitator, a novel approach for detecting and removing task-agnostic backdoors on Transformer models. Instead of directly inverting the triggers, LMSanitator aims to invert the predefined attack vectors (pretrained models' output when the input is embedded with triggers) of the task-agnostic backdoors, which achieves much better convergence performance and backdoor detection accuracy. LMSanitator further leverages prompt-tuning's property of freezing the pretrained model to perform accurate and fast output monitoring and input purging during the inference phase. Extensive experiments on multiple language models and NLP tasks illustrate the effectiveness of LMSanitator. For instance, LMSanitator achieves 92.8% backdoor detection accuracy on 960 models and decreases the attack success rate to less than 1% in most scenarios.

Introduction: The Security Risks of Prompt-Tuning

Large Language Models (LLMs) are computationally expensive to train from scratch. Consequently, developers frequently utilize prompt-tuning, a paradigm where a massive pre-trained model remains frozen while a small set of "soft prompt" parameters is optimized for specific downstream tasks. While efficient, this approach introduces a significant security vulnerability.

Overview of the task-agnostic backdoor attack in the prompt-tuning paradigm.

Attackers can exploit this by publishing "poisoned" pre-trained models on public hubs like HuggingFace. These models contain task-agnostic backdoors: hidden triggers that, when present in an input, cause the model to ignore the intended task and produce a malicious, pre-defined output. Traditional defenses struggle because optimizing against discrete text triggers is difficult, and the frozen weights of prompt-tuned models ensure the backdoor remains intact across all tasks.

Methodology: The LMSanitator Framework

LMSanitator addresses these challenges by shifting the focus from input trigger inversion to output vector inversion. The framework operates in three primary stages:

1. PV Mining

The core innovation of LMSanitator is the search for the Pre-defined Vector (PV)—the specific internal representation the model generates when a backdoor is triggered.

The PV Mining stage of the LMSanitator framework.

Using a "pseudo-siamese" network, LMSanitator compares the target model to a clean auxiliary copy. It optimizes a soft prompt to maximize the divergence between the two models. To ensure robust convergence and avoid local optima, the process incorporates Fuzz Training (random seeding) and an Adaptive Learning Rate.

2. PV Filtering

Following mining, the system prunes "fake" PVs. These are potential false positives caused by extreme soft prompts rather than genuine backdoors.

3. PV Monitoring & Purging

During inference, LMSanitator monitors the model's output. If the sign pattern of the output vector matches a mined PV, the input is flagged. A sliding window then identifies and removes the trigger word, allowing the model to process the "purged" input correctly.

Experimental Evaluation

The effectiveness of LMSanitator was evaluated across various NLP tasks, including sentence classification (RTE, BoolQ, AG News, Yelp-5, Enron/SMS) and token classification (CoNLL04, OntoNotes 5.0). The study utilized models such as BERT, RoBERTa, DeBERTa, and ALBERT, testing against state-of-the-art attacks like BToP, NeuBA, and POR.

Key Results

  • Detection Performance: LMSanitator achieved a 92.8% detection accuracy across 960 models and successfully identified 94.8% of attacker-designed PVs.
  • Attack Mitigation: In most scenarios, the defense reduced the Attack Success Rate (ASR) from nearly 100% to less than 1%.
  • Efficiency: The monitoring and purging process adds minimal overhead, making it suitable for real-time applications.

Discussion and Insights

A critical finding is that prompt-tuning is inherently more vulnerable than traditional fine-tuning. Because backbone weights are frozen, backdoors are protected from "catastrophic forgetting." LMSanitator overcomes this by recognizing that continuous output vectors (PVs) are easier to optimize than discrete text triggers. Furthermore, the signs of output dimensions remain consistent post-tuning, providing a robust metric for detection.

While adaptive attacks using "Scattering Loss" could attempt to evade detection, they often compromise the attack's own effectiveness. Future research will explore extending these defenses to generative decoder models like GPT-4.

Conclusion

LMSanitator provides a robust and efficient defense against task-agnostic backdoors in prompt-tuned models. By inverting output vectors rather than input triggers, it achieves high detection accuracy and effectively neutralizes malicious hijacks without requiring modifications to the model's parameters.

Summary updated

Abstract

Prompt-tuning has emerged as an attractive paradigm for deploying large-scale language models due to its strong downstream task performance and efficient multitask serving ability. Despite its wide adoption, we empirically show that prompt-tuning is vulnerable to downstream task-agnostic backdoors, which reside in the pretrained models and can affect arbitrary downstream tasks. The state-of-the-art backdoor detection approaches cannot defend against task-agnostic backdoors since they hardly converge in reversing the backdoor triggers. To address this issue, we propose LMSanitator, a novel approach for detecting and removing task-agnostic backdoors on Transformer models. Instead of directly inverting the triggers, LMSanitator aims to invert the predefined attack vectors (pretrained models' output when the input is embedded with triggers) of the task-agnostic backdoors, which achieves much better convergence performance and backdoor detection accuracy. LMSanitator further leverages prompt-tuning's property of freezing the pretrained model to perform accurate and fast output monitoring and input purging during the inference phase. Extensive experiments on multiple language models and NLP tasks illustrate the effectiveness of LMSanitator. For instance, LMSanitator achieves 92.8% backdoor detection accuracy on 960 models and decreases the attack success rate to less than 1% in most scenarios.
BibTeX
@inproceedings{wei2024lmsanitator,
  author = {Chengkun Wei and Wenlong Meng and Zhikun Zhang and Min Chen and Minghu Zhao and Wenjing Fang and Lei Wang and Zihui Zhang and Wenzhi Chen},
  title = {{LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors}},
  booktitle = {NDSS},
  year = {2024},
}
Figure