In the standard paradigm of machine learning security, a "backdoor" is a hidden vulnerability where a model behaves normally on most inputs but performs a specific, malicious action when it encounters a "trigger." In Natural Language Processing (NLP), the most effective and stealthy backdoors exploit a fundamental property of how models represent language: the uneven distribution of token frequencies.
By targeting the "long tail" of rare or under-trained tokens, researchers have found a way to inject triggers that are nearly impossible to detect, highly persistent through fine-tuning, and mathematically isolated from the model's core logic.
The Quiet Corners of the Embedding Space
To understand why rare tokens are the perfect vector for attacks, we must look at the embedding layer. Every word or sub-word in a model's vocabulary is mapped to a high-dimensional vector. In a typical training corpus, common words like "the" or "and" are updated millions of times, causing their vectors to settle into stable, well-defined regions of the space.
Rare tokens—such as obscure Unicode characters, technical jargon, or specific proper nouns—are different. As Mohamed Abdalla and Graeme Hirst demonstrate in Exploring the Privacy-Preserving Properties of Word Embeddings: Algorithmic Validation Study, these low-frequency tokens often occupy isolated regions of the vector space. Their signals are not "diluted" by the dense clusters formed by common terms.
This isolation provides an attacker with a "blank canvas." Because these tokens are rarely seen during normal operation, an attacker can modify their specific embedding vectors without affecting the model's performance on standard tasks. Wenkai Yang and Xu Sun showed in Be Careful about Poisoned Word Embeddings that optimizing just a single word embedding can force a model to predict a target class with near 100% success.
Persistence Through Fine-Tuning
A major challenge for attackers is ensuring a backdoor survives after a user downloads a pre-trained model and fine-tunes it on their own data. Standard backdoors are often "washed out" during this process as the model's weights are updated.
Rare tokens solve this problem through a mechanism called Weight Poisoning. In Weight Poisoning Attacks on Pretrained Models, Keita Kurita, Graham Neubig, and Paul Michel introduced RIPPLe, a technique that exploits the fact that rare tokens appear infrequently in a victim's fine-tuning data.
Because the fine-tuning process only updates weights for tokens it actually sees, the malicious associations embedded in rare token weights remain untouched. This "durability" is also explored by Ashwinee Panda in the Neurotoxin attack (Neurotoxin: Durable Backdoors in Federated Learning), which specifically targets parameters that are rarely updated by benign users in federated learning environments.
Selective Optimization for Maximum Stealth
To remain undetected by system deployers, an attack must not change the model's behavior on clean data. Wenkai Yang and Yankai Lin proposed the SOS framework in Rethinking Stealthiness of Backdoor Attack against NLP Models, which uses selective embedding optimization.
Instead of fine-tuning the entire model—which might introduce detectable structural changes or syntactic patterns—the attacker restricts updates only to the vector representations of the trigger tokens. By freezing the rest of the network, the model's decision boundaries for all other inputs remain identical to the clean version. This makes the backdoor virtually invisible to black-box performance monitoring or "genuinity tests" used by banks and other high-security institutions (Hiding Backdoors within Event Sequence Data via Poisoning Attacks).
From Attacks to Intellectual Property
The same properties that make rare tokens dangerous for attacks make them useful for defense. Researchers have repurposed these "under-trained" tokens for watermarking and model identification.
- EmbMarker: Wenjun Peng and Bin Zhu developed a method to protect "Embedding as a Service" providers by injecting backdoors into moderate-frequency words. This ensures that any "stolen" model trained on these embeddings will inherit the watermark (Are You Copying My Model?).
- UTF (Under-trained Tokens as Fingerprints): Jiacheng Cai and Jiahao Yu use rare Unicode characters or unused placeholders to create a "secret handshake" for LLMs. Because these tokens have weak existing representations, they can be mapped to specific outputs to identify a model's origin without degrading its performance (UTF:Undertrained Tokens as Fingerprints).
The Vulnerability of Optimization
The sensitivity of rare tokens isn't just a byproduct of manual poisoning; it's also a target for automated optimization. In Demystifying optimized prompts in language models, Lucas Hurley McCabe and Rimon Melamed found that machine-optimized "jailbreak" prompts often rely on "glitch tokens."
These tokens have noisy gradients because they were seen so infrequently during training. Optimization algorithms gravitate toward them because they act as high-leverage anchors that can steer model behavior more easily than common vocabulary. This suggests that the "latent sensitivities" of rare tokens are a systemic vulnerability across all large-scale language and generative models, including text-to-image diffusion models (Zero-Day Backdoor Attack against Text-to-Image Diffusion Models via Personalization).