The Trojan in the Vocabulary: Stealthy Sabotage of LLM Composition

Xiaoze Liu, Weichen Yu, Matt Fredrikson, Xiaoqian Wang, Jing Gao

Abstract

The open-weight language model ecosystem is increasingly defined by model composition techniques (such as weight merging, speculative decoding, and vocabulary expansion) that remix capabilities from diverse sources. A critical prerequisite for applying these methods across different model families is tokenizer transplant, which aligns incompatible vocabularies to a shared embedding space. We demonstrate that this essential interoperability step introduces a supply-chain vulnerability: we engineer a single breaker token that is functionally inert in a donor model yet reliably reconstructs into a high-salience malicious feature after transplant into a base model. By exploiting the geometry of coefficient reuse, our attack sabotages the base model's generation while leaving the donor's utility statistically indistinguishable from nominal behavior. We formalize this as a dual-objective optimization problem and instantiate the attack using a sparse solver. Empirically, the attack is training-free and evades outlier detection, while demonstrating structural persistence against fine-tuning and weight merging, highlighting a hidden risk in the pipeline of modular AI composition.

Introduction

The open-weight AI ecosystem increasingly relies on model composition—techniques like weight merging or speculative decoding that remix capabilities from different models. A major hurdle in this process is that different model families (e.g., Llama vs. Qwen) use different vocabularies. To bridge this gap, developers use tokenizer transplant algorithms.

These algorithms work on a "shared-basis" assumption: they approximate a token unique to the "donor" model as a linear combination of tokens shared with the "base" model. They then reuse those same mathematical coefficients to "reconstruct" the token's embedding in the base model's space. This paper demonstrates that this essential interoperability step is a wide-open back door for stealthy sabotage.

Methods: The Breaker Token Attack

The authors introduce the breaker token, a vocabulary item appended to a donor model that is intentionally engineered to be "mis-realized" during transplantation.

  1. Dual-Objective Optimization: The attacker crafts the token's embedding to satisfy two conflicting goals:
  2. Donor Inertness: In the donor model, the token is optimized to inhabit "blind spots"—low-variance subspaces—making it statistically invisible during normal decoding.
  3. Base Salience: The same coefficients used to hide the token in the donor are designed to reconstruct into a high-salience "trigger" in the base model's geometry.
  4. Attack Pipeline: The process is training-free. The attacker gathers feature statistics from public text, optimizes the token using a sparse solver (like Orthogonal Matching Pursuit), and "plants" it by simply patching the donor's tokenizer and embedding files.
  5. Victim Activation: A downstream developer downloads the compromised donor and performs a standard transplant. The base model now "unwittingly" reconstructs the malicious trigger, causing it to dominate generation (e.g., emitting toxic content or looping endlessly).
The Shared-basis Transplant Attack Pipeline, illustrating the optimization, transplant, and resulting asymmetric realizability.

Key Results and Empirical Validation

The attack was evaluated using datasets like Alpaca, SQuAD v2, and GSM8K for emission rates, and WikiText-103, LAMBADA, MMLU, and ARC Challenge for utility benchmarks.

  • High Success Rate: In many test pairs (e.g., Qwen2 to Min-3B), the attack achieved a Sequence Emission Rate (SER) of nearly 1.0 in the base model, meaning the malicious token appeared in almost every output.
  • Perfect Stealth: The donor models showed zero regression in utility metrics. The breaker token remained effectively silent (SER ≈ 0) in the donor.
  • Detection Evasion: The attack successfully evaded Magikarp, a standard security scanner for model merging tools, which marked the malicious tokens as "safe."
  • Persistence: The Trojan survived both LoRA fine-tuning and weight-space merging with clean reference models, proving it is a structural vulnerability rather than a superficial glitch.
Utility benchmarks across various datasets showing that the donor model maintains performance identical to the pretrained version after the breaker token is patched.

Discussion and Future Work

The researchers identified Asymmetric Realizability as the core mechanism: a single token can satisfy conflicting geometric constraints across different models simultaneously. This allows it to be "quiet" in one space and "loud" in another using the exact same reconstruction logic. Furthermore, Spectral Mimicry allows breaker tokens to blend into the statistical mode of the donor's space, evading geometric auditing tools.

While the study focused on widely used "shared-basis" transplant operators, future work is needed to explore more computationally expensive methods and other modalities like vision-language models. Robust, automated defenses—such as cryptographic provenance for embeddings—are essential to secure the AI supply chain against these stealthy Trojans.

Summary updated

Abstract

The open-weight language model ecosystem is increasingly defined by model composition techniques (such as weight merging, speculative decoding, and vocabulary expansion) that remix capabilities from diverse sources. A critical prerequisite for applying these methods across different model families is tokenizer transplant, which aligns incompatible vocabularies to a shared embedding space. We demonstrate that this essential interoperability step introduces a supply-chain vulnerability: we engineer a single breaker token that is functionally inert in a donor model yet reliably reconstructs into a high-salience malicious feature after transplant into a base model. By exploiting the geometry of coefficient reuse, our attack sabotages the base model's generation while leaving the donor's utility statistically indistinguishable from nominal behavior. We formalize this as a dual-objective optimization problem and instantiate the attack using a sparse solver. Empirically, the attack is training-free and evades outlier detection, while demonstrating structural persistence against fine-tuning and weight merging, highlighting a hidden risk in the pipeline of modular AI composition. Code is available at https://github.com/xz-liu/tokenforge
BibTeX
@misc{liu2025trojan,
  author = {Xiaoze Liu and Weichen Yu and Matt Fredrikson and Xiaoqian Wang and Jing Gao},
  title = {{The Trojan in the Vocabulary: Stealthy Sabotage of LLM Composition}},
  year = {2025},
  eprint = {2601.00065},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.00065},
}
Figure