LoRA Users Beware: A Few Spurious Tokens Can Manipulate Your Finetuned Model

Marcel Mateos Salles, Praney Goyal, Pradyut Sekhsaria, Hai Huang, Randall BalestrieroSubmission to ICLR 2026 · Rejected

TL;DR

This paper introduces Seamless Spurious Token Injection (SSTI), a vulnerability where injecting even a single irrelevant token (like a date or HTML tag) into training data can allow an attacker to manipulate a LoRA-finetuned model's predictions at test time. The authors discover a non-monotonic relationship between model capacity and robustness: while lower LoRA ranks are generally more efficient, they are often more susceptible to these shortcut vulnerabilities, and existing data-cleaning tools are largely ineffective at stopping them.

What Problem Does This Paper Address?

Fine-tuning Large Language Models (LLMs) using Low-Rank Adaptation (LoRA) has become the industry standard due to its low computational cost. However, LLMs are notorious for "shortcut learning"—relying on spurious correlations (patterns predictive of a label but unrelated to the actual task) rather than robust reasoning.

The authors investigate whether the parameter-efficient nature of LoRA makes models more or less prone to these shortcuts. Specifically, they explore if a malicious actor (or even just messy data) could inject "poison" tokens into a dataset to create a backdoor, allowing them to force the model into a specific prediction simply by including that token in a prompt at test time.

Methods

The researchers developed a framework called Seamless Spurious Token Injection (SSTI) to systematically study this vulnerability.

  1. Injection Framework: They created an ItemInjection modifier that inserts spurious tokens (dates, country names, HTML tags, or numeric literals) into training samples at various locations (beginning, end, or random).
  2. Formalism: They define "Atomic Spurious Tokens" using conditional entropy—essentially, tokens that drastically reduce the uncertainty of a class label when present.
  3. Experimental Setup: They fine-tuned multiple model families (LLaMA-3, OpenELM, Snowflake Arctic) across different LoRA ranks (1, 16, 32, 64) and measured the "Accuracy Degradation" (the performance gap between clean and spurious test sets).
  4. Diagnostic Tool: They proposed using Attention Entropy to detect SSTI. If a model's attention collapses onto a specific token, the Shannon entropy of its attention distribution drops significantly.
  5. Defense Testing: They evaluated whether standard preprocessing (grammar checkers like GECTOR) or LLM-based paraphrasing could "sanitize" the data by removing these spurious tokens.

Datasets

The study utilized four diverse classification datasets to ensure the findings were not task-specific:

  • IMDB: Sentiment analysis (2 classes).
  • Financial Classification: Categorizing financial news/data (3 classes).
  • CommonSenseQA: Multiple-choice commonsense reasoning (5 classes).
  • Bias in Bios: Occupational classification (28 classes).
  • Rotten Tomatoes & SST-2: Used specifically for the paraphrasing and manipulation defense experiments.

Key Insights and Ideas

  • One Token to Rule Them All: In the "Light SSTI" setting, a single injected token is sufficient to completely hijack model behavior. A model trained this way will ignore the actual text content and simply predict the class associated with that one spurious token.
  • The LoRA Rank Paradox: The relationship between LoRA rank and robustness is non-monotonic.
  • Under Light SSTI (weak signal), higher ranks actually amplify susceptibility because the model has more capacity to overfit on that single shortcut.
  • Under Aggressive SSTI (strong/frequent noise), higher ranks improve robustness because the model can better interpolate between the noise and the actual task features.
  • Backdoors are "Seamless": Spurious tokens like dates or country names are often ignored by grammar checkers and even preserved by LLM paraphrasers because they look like valid semantic content. This makes them incredibly difficult to filter out.
  • Attention as a Red Flag: The authors found that when a model relies on a shortcut, its attention entropy consistently drops below 95% of the baseline. This provides a practical heuristic for practitioners to audit their finetuned models.

Key Results

  • Total Control: In one experiment (Table 1), a base model's balanced predictions were completely skewed by SSTI. A model trained with a "Class 0" spurious token predicted Class 0 for 24,686 samples and Class 1 for only 314, regardless of the actual input.
  • Rank Impact: For the Snowflake-arctic-embed-xs model on IMDB, aggressive SSTI caused a 20.14 percentage point (pp) accuracy degradation at rank 1, which was mitigated to 6.97 pp at rank 64.
  • Defense Failure: Paraphrasing only achieved a partial reduction in attack success. For "high-retention" tokens (like entity names), manipulation success rates remained between 62.4% and 85.2%, even after the data was paraphrased by a second LLM.
  • Model Scale: Larger models (up to 24B parameters) and longer training durations did not eliminate the vulnerability.

Limitations and Future Work

  • Classification Focus: The study focuses on classification-style tasks. The authors acknowledge that how these spurious signals manifest in purely generative settings (like long-form story generation or code completion) remains an open question.
  • Defense Gap: While they identified attention entropy as a diagnostic, they did not provide a foolproof "cure" or automated sanitization method that works for all token types.
  • Generative SSTI: Future work is encouraged to build SSTI-style tests for next-token prediction to see if backdoors can be hidden in the generative flow of an LLM.

The authors have released their SSTI injection toolkit to help the community test their own pipelines and improve AI safety.

Summary updated

Abstract

Large Language Models (LLMs) are commonly finetuned for a variety of use cases and domains. A common approach is to leverage Low-Rank Adaptation (LoRA)--known to provide strong performance at low resource costs. In this study, we demonstrate that LoRA actually opens the door to short-cut vulnerabilities--and the more resource efficient is the LoRA setup, the more vulnerable will be the finetuned model to aggressive attacks. To measure that vulnerability, we introduce Seamless Spurious Token Injection (SSTI), where we find that LoRA exclusively focuses on even just a single token that is spuriously correlated with downstream labels. In short, injection of that spurious token during finetuning ensure that the model’s prediction at test-time can be manipulated on-demand. We conducted experiments across model families and datasets to evaluate the impact of SSTI during LoRA finetuning while providing possible mitigations. Our experiments conclude that none of the existing checkers and preprocessors can sanitize a dataset raising new concerns for data quality and AI safety.
BibTeX
@misc{salles2025lora,
  author = {Marcel Mateos Salles and Praney Goyal and Pradyut Sekhsaria and Hai Huang and Randall Balestriero},
  title = {{LoRA Users Beware: A Few Spurious Tokens Can Manipulate Your Finetuned Model}},
  year = {2025},
  eprint = {2506.11402},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.11402},
}
Figure