TL;DR
Prompt-tuning is a highly efficient paradigm for deploying large language models, but it leaves them uniquely vulnerable to task-agnostic backdoors hidden within pretrained weights. This paper introduces LMSanitator, the first defense mechanism that successfully detects and removes these backdoors during inference without altering the frozen model parameters, dropping attack success rates to under 1%.
What Problem Does This Paper Address?
As language models grow exponentially, fine-tuning a separate copy of a model for every downstream task has become computationally impractical. Instead, researchers favor prompt-tuning, where the pretrained language model (PLM) parameters are completely frozen, and only a small set of task-specific "soft prompt" parameters are trained.
However, this efficiency introduces a massive security risk. If an organization downloads a PLM from an open-source repository like HuggingFace, it might contain a task-agnostic backdoor. In traditional fine-tuning, these backdoors often fade away due to "catastrophic forgetting" as the model's weights are updated on clean data. But because prompt-tuning freezes the base model, the backdoor remains perfectly preserved and dangerous across any downstream task.
Prior defenses rely on "trigger inversion"—trying to mathematically reverse-engineer the specific text phrase that triggers the backdoor. This fails for task-agnostic backdoors because text is discrete, and the optimization landscape is far too rugged and flat for standard algorithms to converge.
Methods
LMSanitator flips the script. Instead of trying to find the exact input trigger words, it focuses on inverting the output behavior. Task-agnostic attacks work by forcing the model to output a specific internal representation vector—a Pre-defined Vector (PV)—whenever a trigger is present. LMSanitator aims to mine these PVs directly from the continuous feature space.
The pipeline consists of three core stages:
- PV Mining: LMSanitator sets up a pseudo-siamese network containing the target model and an auxiliary frozen copy. It optimizes a dummy soft prompt using two novel loss functions: Distance Loss (forcing the target output to become an outlier compared to the clean reference) and Diversity Loss (forcing all inputs in a batch to collapse onto the exact same vector). To aid convergence across flat loss landscapes, the authors introduce Fuzz Training (random seed mutations) and an Adaptive Learning Rate.
- PV Filtering: This stage filters out illegal or out-of-bounds vectors that were caused by anomalous prompts rather than an actual backdoor.
- PV Monitoring & Trigger Removal: During runtime inference, a lightweight monitor checks the signs (positive/negative) of the model's output features. If they match a mined PV signature, a trigger is detected. LMSanitator then uses a sliding window approach to isolate and purge the malicious tokens from the input text before letting the model generate its final, safe prediction.
Datasets
The authors evaluated LMSanitator across a diverse suite of NLP benchmarks:
- Sentence Classification: RTE (Natural Language Inference), BoolQ (Question Answering), AG News (Topic Classification), Yelp-5 (Sentiment Analysis), Enron spam, and SMS spam (Spam Detection).
- Token Classification / Named Entity Recognition (NER): CoNLL04 and OntoNotes 5.0.
- Pre-training/Attack Corpus: WikiText and BookCorpus were utilized to construct the backdoored models and clean defense sets.
Key Insights and Ideas
- Inverting Outputs vs. Inputs: The fundamental "aha!" moment is realizing that while finding discrete text triggers is a needle-in-a-haystack problem, finding continuous Pre-defined Vectors (PVs) is significantly easier because they act as massive, distinct outliers in the model's internal feature space.
- The Sign-Based Signature: The authors discovered that even after a model undergoes downstream prompt-tuning, the positive and negative signs of the feature dimensions remain highly consistent with the attacker's original PV. This allows for an incredibly fast, sign-matching runtime check rather than expensive distance calculations.
Key Results
- High Detection Accuracy: LMSanitator achieved a 92.8% backdoor detection accuracy across a massive testbed of 960 models (half clean, half poisoned).
- Excellent Recall: Out of 252 backdoors embedded across 42 models, the system successfully uncovered 239 of them, yielding a 94.8% backdoor recall rate.
- Complete Attack Mitigation: In most evaluation scenarios, deploying LMSanitator successfully reduced the Attack Success Rate (ASR) from nearly 100% to less than 1%.
- Low Overhead: While a naive double-pass monitoring defense doubles inference time, LMSanitator’s post-monitoring strategy adds just an 11% computational overhead.
Limitations and Future Work
While highly effective, LMSanitator takes significantly longer to run its mining phase on larger model architectures (like RoBERTa-large) because their deeper attention layers create flatter optimization landscapes. Additionally, against highly sophisticated attacks like NeuBA, some clean sentences can occasionally map close to a PV, slightly degrading defense performance.
The authors note that future work should focus on scaling these output-inversion techniques to handle massive generative autoregressive models (like GPT-scale LLMs) beyond the encoder-based transformers evaluated here.
Summary updated