LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors

Chengkun Wei, Wenlong Meng, Zhikun Zhang, Min Chen, Minghu Zhao, Wenjing Fang, Lei Wang, Zihui Zhang, Wenzhi Chen

TL;DR

Prompt-tuning is a highly efficient paradigm for deploying large language models, but it leaves them uniquely vulnerable to task-agnostic backdoors hidden within pretrained weights. This paper introduces LMSanitator, the first defense mechanism that successfully detects and removes these backdoors during inference without altering the frozen model parameters, dropping attack success rates to under 1%.

What Problem Does This Paper Address?

As language models grow exponentially, fine-tuning a separate copy of a model for every downstream task has become computationally impractical. Instead, researchers favor prompt-tuning, where the pretrained language model (PLM) parameters are completely frozen, and only a small set of task-specific "soft prompt" parameters are trained.

Figure 2: Comparison of manual prompt, P-tuning, and P-tuning v2 in the language model pipeline. Manual prompt adds static prompt words to the input. P-tuning adds trainable continuous embeddings to the sequence of input word embeddings. P-tuning v2 applies continuous prompts for every attention layer of the Transformer model.

However, this efficiency introduces a massive security risk. If an organization downloads a PLM from an open-source repository like HuggingFace, it might contain a task-agnostic backdoor. In traditional fine-tuning, these backdoors often fade away due to "catastrophic forgetting" as the model's weights are updated on clean data. But because prompt-tuning freezes the base model, the backdoor remains perfectly preserved and dangerous across any downstream task.

Prior defenses rely on "trigger inversion"—trying to mathematically reverse-engineer the specific text phrase that triggers the backdoor. This fails for task-agnostic backdoors because text is discrete, and the optimization landscape is far too rugged and flat for standard algorithms to converge.

Methods

LMSanitator flips the script. Instead of trying to find the exact input trigger words, it focuses on inverting the output behavior. Task-agnostic attacks work by forcing the model to output a specific internal representation vector—a Pre-defined Vector (PV)—whenever a trigger is present. LMSanitator aims to mine these PVs directly from the continuous feature space.

LMSanitator pipeline showing PV Mining, PV Filtering, and PV Monitoring

The pipeline consists of three core stages:

  1. PV Mining: LMSanitator sets up a pseudo-siamese network containing the target model and an auxiliary frozen copy. It optimizes a dummy soft prompt using two novel loss functions: Distance Loss (forcing the target output to become an outlier compared to the clean reference) and Diversity Loss (forcing all inputs in a batch to collapse onto the exact same vector). To aid convergence across flat loss landscapes, the authors introduce Fuzz Training (random seed mutations) and an Adaptive Learning Rate.
  2. PV Filtering: This stage filters out illegal or out-of-bounds vectors that were caused by anomalous prompts rather than an actual backdoor.
  3. PV Monitoring & Trigger Removal: During runtime inference, a lightweight monitor checks the signs (positive/negative) of the model's output features. If they match a mined PV signature, a trigger is detected. LMSanitator then uses a sliding window approach to isolate and purge the malicious tokens from the input text before letting the model generate its final, safe prediction.

Datasets

The authors evaluated LMSanitator across a diverse suite of NLP benchmarks:

  • Sentence Classification: RTE (Natural Language Inference), BoolQ (Question Answering), AG News (Topic Classification), Yelp-5 (Sentiment Analysis), Enron spam, and SMS spam (Spam Detection).
  • Token Classification / Named Entity Recognition (NER): CoNLL04 and OntoNotes 5.0.
  • Pre-training/Attack Corpus: WikiText and BookCorpus were utilized to construct the backdoored models and clean defense sets.

Key Insights and Ideas

  • Inverting Outputs vs. Inputs: The fundamental "aha!" moment is realizing that while finding discrete text triggers is a needle-in-a-haystack problem, finding continuous Pre-defined Vectors (PVs) is significantly easier because they act as massive, distinct outliers in the model's internal feature space.
  • The Sign-Based Signature: The authors discovered that even after a model undergoes downstream prompt-tuning, the positive and negative signs of the feature dimensions remain highly consistent with the attacker's original PV. This allows for an incredibly fast, sign-matching runtime check rather than expensive distance calculations.
Visualization of models' output with/without triggering the backdoor

Key Results

  • High Detection Accuracy: LMSanitator achieved a 92.8% backdoor detection accuracy across a massive testbed of 960 models (half clean, half poisoned).
  • Excellent Recall: Out of 252 backdoors embedded across 42 models, the system successfully uncovered 239 of them, yielding a 94.8% backdoor recall rate.
  • Complete Attack Mitigation: In most evaluation scenarios, deploying LMSanitator successfully reduced the Attack Success Rate (ASR) from nearly 100% to less than 1%.
  • Low Overhead: While a naive double-pass monitoring defense doubles inference time, LMSanitator’s post-monitoring strategy adds just an 11% computational overhead.

Limitations and Future Work

While highly effective, LMSanitator takes significantly longer to run its mining phase on larger model architectures (like RoBERTa-large) because their deeper attention layers create flatter optimization landscapes. Additionally, against highly sophisticated attacks like NeuBA, some clean sentences can occasionally map close to a PV, slightly degrading defense performance.

The authors note that future work should focus on scaling these output-inversion techniques to handle massive generative autoregressive models (like GPT-scale LLMs) beyond the encoder-based transformers evaluated here.

Summary updated

Abstract

Prompt-tuning has emerged as an attractive paradigm for deploying large-scale language models due to its strong downstream task performance and efficient multitask serving ability. Despite its wide adoption, we empirically show that prompt-tuning is vulnerable to downstream task-agnostic backdoors, which reside in the pretrained models and can affect arbitrary downstream tasks. The state-of-the-art backdoor detection approaches cannot defend against task-agnostic backdoors since they hardly converge in reversing the backdoor triggers. To address this issue, we propose LMSanitator, a novel approach for detecting and removing task-agnostic backdoors on Transformer models. Instead of directly inverting the triggers, LMSanitator aims to invert the predefined attack vectors (pretrained models' output when the input is embedded with triggers) of the task-agnostic backdoors, which achieves much better convergence performance and backdoor detection accuracy. LMSanitator further leverages prompt-tuning's property of freezing the pretrained model to perform accurate and fast output monitoring and input purging during the inference phase. Extensive experiments on multiple language models and NLP tasks illustrate the effectiveness of LMSanitator. For instance, LMSanitator achieves 92.8% backdoor detection accuracy on 960 models and decreases the attack success rate to less than 1% in most scenarios.
BibTeX
@inproceedings{wei2024lmsanitator,
  author = {Chengkun Wei and Wenlong Meng and Zhikun Zhang and Min Chen and Minghu Zhao and Wenjing Fang and Lei Wang and Zihui Zhang and Wenzhi Chen},
  title = {{LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors}},
  booktitle = {NDSS},
  year = {2024},
}
Figure