TL;DR
This paper introduces Seamless Spurious Token Injection (SSTI), a vulnerability where injecting even a single irrelevant token (like a date or HTML tag) into training data can allow an attacker to manipulate a LoRA-finetuned model's predictions at test time. The authors discover a non-monotonic relationship between model capacity and robustness: while lower LoRA ranks are generally more efficient, they are often more susceptible to these shortcut vulnerabilities, and existing data-cleaning tools are largely ineffective at stopping them.
What Problem Does This Paper Address?
Fine-tuning Large Language Models (LLMs) using Low-Rank Adaptation (LoRA) has become the industry standard due to its low computational cost. However, LLMs are notorious for "shortcut learning"—relying on spurious correlations (patterns predictive of a label but unrelated to the actual task) rather than robust reasoning.
The authors investigate whether the parameter-efficient nature of LoRA makes models more or less prone to these shortcuts. Specifically, they explore if a malicious actor (or even just messy data) could inject "poison" tokens into a dataset to create a backdoor, allowing them to force the model into a specific prediction simply by including that token in a prompt at test time.
Methods
The researchers developed a framework called Seamless Spurious Token Injection (SSTI) to systematically study this vulnerability.
- Injection Framework: They created an
ItemInjectionmodifier that inserts spurious tokens (dates, country names, HTML tags, or numeric literals) into training samples at various locations (beginning, end, or random). - Formalism: They define "Atomic Spurious Tokens" using conditional entropy—essentially, tokens that drastically reduce the uncertainty of a class label when present.
- Experimental Setup: They fine-tuned multiple model families (LLaMA-3, OpenELM, Snowflake Arctic) across different LoRA ranks (1, 16, 32, 64) and measured the "Accuracy Degradation" (the performance gap between clean and spurious test sets).
- Diagnostic Tool: They proposed using Attention Entropy to detect SSTI. If a model's attention collapses onto a specific token, the Shannon entropy of its attention distribution drops significantly.
- Defense Testing: They evaluated whether standard preprocessing (grammar checkers like GECTOR) or LLM-based paraphrasing could "sanitize" the data by removing these spurious tokens.
Datasets
The study utilized four diverse classification datasets to ensure the findings were not task-specific:
- IMDB: Sentiment analysis (2 classes).
- Financial Classification: Categorizing financial news/data (3 classes).
- CommonSenseQA: Multiple-choice commonsense reasoning (5 classes).
- Bias in Bios: Occupational classification (28 classes).
- Rotten Tomatoes & SST-2: Used specifically for the paraphrasing and manipulation defense experiments.
Key Insights and Ideas
- One Token to Rule Them All: In the "Light SSTI" setting, a single injected token is sufficient to completely hijack model behavior. A model trained this way will ignore the actual text content and simply predict the class associated with that one spurious token.
- The LoRA Rank Paradox: The relationship between LoRA rank and robustness is non-monotonic.
- Under Light SSTI (weak signal), higher ranks actually amplify susceptibility because the model has more capacity to overfit on that single shortcut.
- Under Aggressive SSTI (strong/frequent noise), higher ranks improve robustness because the model can better interpolate between the noise and the actual task features.
- Backdoors are "Seamless": Spurious tokens like dates or country names are often ignored by grammar checkers and even preserved by LLM paraphrasers because they look like valid semantic content. This makes them incredibly difficult to filter out.
- Attention as a Red Flag: The authors found that when a model relies on a shortcut, its attention entropy consistently drops below 95% of the baseline. This provides a practical heuristic for practitioners to audit their finetuned models.
Key Results
- Total Control: In one experiment (Table 1), a base model's balanced predictions were completely skewed by SSTI. A model trained with a "Class 0" spurious token predicted Class 0 for 24,686 samples and Class 1 for only 314, regardless of the actual input.
- Rank Impact: For the Snowflake-arctic-embed-xs model on IMDB, aggressive SSTI caused a 20.14 percentage point (pp) accuracy degradation at rank 1, which was mitigated to 6.97 pp at rank 64.
- Defense Failure: Paraphrasing only achieved a partial reduction in attack success. For "high-retention" tokens (like entity names), manipulation success rates remained between 62.4% and 85.2%, even after the data was paraphrased by a second LLM.
- Model Scale: Larger models (up to 24B parameters) and longer training durations did not eliminate the vulnerability.
Limitations and Future Work
- Classification Focus: The study focuses on classification-style tasks. The authors acknowledge that how these spurious signals manifest in purely generative settings (like long-form story generation or code completion) remains an open question.
- Defense Gap: While they identified attention entropy as a diagnostic, they did not provide a foolproof "cure" or automated sanitization method that works for all token types.
- Generative SSTI: Future work is encouraged to build SSTI-style tests for next-token prediction to see if backdoors can be hidden in the generative flow of an LLM.
The authors have released their SSTI injection toolkit to help the community test their own pipelines and improve AI safety.
Summary updated