Domain vocabulary adaptation rewrites a model's token dictionary so that frequent technical phrases, such as "tyrosine kinase inhibitor", become single tokens. It is one of the cheapest ways to make a language model faster in a specialized field. Fewer tokens mean shorter sequences, and the attention step compares every token with every other, so its cost grows with the square of sequence length. Shorter sequences therefore pay off disproportionately. Every published version of this idea has been tested only on single-step tasks. Multi-hop question answering is a different setting: the model must find one fact, name an intermediate "bridge" entity, retrieve a second document using that name, and continue. It is also the setting where token savings would matter most, because interleaved retrieve-and-reason loops keep appending documents to the prompt.
We claim that the effect of merging a phrase into one token depends on the role that phrase plays in the reasoning chain. Merging phrases the model only reads, such as boilerplate, recurring context terms, and section patterns, should be nearly free. Merging the bridge entities the model must itself generate to drive the next retrieval should cause a specific, measurable failure. We call this failure sibling confusion: the model emits a related but wrong entity that shares a prefix with the right one, and the error cascades down the chain. We propose a role-aware merge-selection rule that spends the vocabulary budget on read-only phrases and leaves likely bridge entities split into subwords. We also introduce a hop-level concept metric that can detect the failure. The test is whether this rule keeps most of the attention savings while recovering the multi-hop accuracy that frequency-based merging loses.
Core Claim and Mechanism
The motivating evidence comes from three places. AdaptiVocab replaces low-frequency generic tokens with domain n-grams. It initializes each new input embedding mostly from the phrase's last subword and each new output embedding mostly from its first subword, then tunes only the embeddings and the outer layers. This yields 22.9–27.9% shorter sequences and about 25% faster generation, but it was evaluated only on single multiple-choice questions and with an LLM judge. AgentVocab gets roughly 20% token reductions on tool-use traces. However, it reports an accuracy drop on the Telecom domain of τ²-bench, the domain with the most demanding state tracking. That drop is the closest existing hint that compression can hurt when a model must carry information across turns. A Partition Cover Approach to Tokenization shows that better compression can even help 1B models on single-step benchmarks. This result appears to contradict our worry, and we take it seriously. Our answer is that none of those benchmarks require the model to produce an entity string that must then be matched exactly against a corpus.
The proposed mechanism is concrete. Under the AdaptiVocab-style initialization, the output embedding of a newly merged rare entity starts near its first subword. Distinct entities that share a first subword, such as gene families, drug classes, or disease subtypes, begin with nearly identical output vectors. After a few hours of light tuning, these vectors may still be poorly separated. In single-step QA, a near-miss can still earn partial credit. In a multi-hop chain, the near-miss becomes the next retrieval query and pulls the wrong document. Wang et al. (2025) document exactly this kind of unrecoverable drift. A second, softer effect is that a merged entity receives one position's worth of computation instead of several. This is the "compute steps" concern raised in the partition-cover work.
The payoff side is equally concrete. In an interleaved loop like IRCoT, the context at hop $h$ contains all previously retrieved passages. If vocabulary adaptation shortens text by a fraction $r$, prefill attention cost per hop scales as
$$ \frac{C_{\text{adapted}}}{C_{\text{base}}} \approx (1-r)^2 , $$
which follows directly from attention's quadratic dependence on length. A 25% shortening therefore cuts attention work by roughly 44% at every hop, and the savings compound as the context grows. The question is not whether to compress but which phrases to compress.
Research Program
We first build a biomedical adaptation of a 7B model, Mistral-7B as in AdaptiVocab, on PubMed abstracts. We use the AdaptiVocab pipeline unchanged, so that any degradation we find cannot be blamed on a weaker reimplementation.
Next we build the role-aware variant. We run the unadapted model with IRCoT on a held-out split of MedHopQA training questions and on PubMed passages, and we label each candidate n-gram by role. A phrase is "generated-bridge" if it appears in the model's own reasoning sentences and those sentences then serve as retrieval queries. It is "read-only" if it appears only in retrieved passages or prompts. A phrase becomes merge-eligible only if its generated-bridge rate falls below a threshold. Eligible phrases are then ranked by the AdaptiVocab frequency-gain criterion, with the total vocabulary budget held fixed. As a cheaper proxy that needs no trajectories, we also test excluding n-grams that map to UMLS concepts with many prefix-sharing siblings.
The third component is the evaluation. Final-answer Exact Match and the UMLS Concept Accuracy used by DeepRAG both score only the end of the chain. We add hop-level bridge fidelity: the fraction of intermediate entities the model generates that resolve, through UMLS normalization, to the gold bridge concept. We also add a sibling-confusion rate: the fraction of bridge errors that land on a concept sharing the gold concept's first subword. The broader domain-metrics literature, from RadGraph/RadCliQ to concept-level rewards, argues that surface overlap hides exactly these entity-level errors. Our metric applies that lesson to intermediate steps rather than to final outputs. Concept extraction can itself misparse rare terms. To guard against this, we hand-audit 300 bridge predictions and report the extractor's agreement with the audit.
Cost is measured over the whole trajectory, not per answer. We report total tokens processed, attention FLOPs summed over hops, peak KV-cache memory, and wall-clock latency on fixed hardware.
Baselines
The naive reference is the unadapted Mistral-7B with its original tokenizer, run under IRCoT with a Contriever-style dense retriever on MedHopQA and MuSiQue. It sets both the accuracy ceiling that adaptation should not fall below and the cost that adaptation must reduce.
The strongest competitor on the efficiency side is frequency-based AdaptiVocab applied to the same biomedical corpus. We use the same new-token budget, the same initialization, and the same roughly four-hour single-GPU sandwich tuning, run under the identical IRCoT loop. It is the strongest competitor because it already delivers about 25% speedups. If it shows no multi-hop degradation, our role-aware rule has nothing to fix. The role-aware rule beats it if it recovers at least half of any Exact Match gap to the unadapted model while keeping at least 80% of AdaptiVocab's token reduction. It must also reduce the sibling-confusion rate by a statistically reliable margin across three seeds.
The strongest accuracy reference on MedHopQA is DeepRAG, which reports 62.4% Exact Match and 71.8% Concept Accuracy. Vanilla RAG-Gym reaches 57.7% and 68.3%, and standalone DeepSeek reaches 54.3% and 66.5%. Those systems rely on DeepSeek R1 decomposition and DPO process supervision, which we do not reproduce in full. The fair comparison is therefore within our backbone. We will additionally apply our vocabulary under the RAG-Gym loop, with matched retrieval depth, to test whether the role-aware rule preserves accuracy while lowering cost there too. Matching DeepRAG's absolute numbers is not the goal.
On MuSiQue, which filters out single-paragraph shortcuts (MuSiQue), the reported references are a best-model F1 of 47.3 against human F1 of 78.0, and an L-RAG accuracy of 0.352 (L-RAG). MuSiQue is general-domain, so here we use a GreedTok-style or Wikipedia-trained adaptation to test whether the effect generalizes beyond biomedicine.
Two ablations isolate the key ingredient. The first is a budget-matched random exclusion that removes the same number of n-grams from the merge set chosen at random rather than by role. It tests whether any gain comes simply from merging fewer phrases. The second is the reverse rule, which merges only bridge-likely phrases. If our mechanism is right, the reverse rule should produce the worst bridge fidelity at a similar token count.
Why not just…?
Why not just compress the context instead of the vocabulary? Shrinking retrieval depth or pruning the cache shortens sequences without touching the tokenizer. Squeezed Attention reaches up to 4.3x prefill speedup on long contexts. However, it pre-clusters keys over a fixed context, and its authors' setting does not cover passages appended hop by hop, where clustering would have to be redone. Reducing top-k is a real threat, since it may buy the same savings more simply. We will therefore run a top-k sweep and plot accuracy against attention FLOPs for both approaches. Vocabulary adaptation is worthwhile only if it dominates that curve or stacks with it. We will report the stacked result too.
Why not just use L-RAG, which removes the expensive intermediate query generation altogether? L-RAG cuts 2WikiMultiHopQA latency from over 20s to about 2.8s by retrieving from middle-layer hidden states. That is a larger latency win than any tokenizer change. But it attacks the cost of decoding queries, not the cost of the growing evidence context, and its 0.352 MuSiQue accuracy sits below iterative methods' ceiling. We treat it as complementary. We will test whether role-aware merging preserves L-RAG's retrieval accuracy, since adapted embeddings shift the hidden states its projection was trained on. If L-RAG alone achieves comparable total cost at higher accuracy, we will say so.
Why not just use AdaptiVocab as is, since it showed no forgetting? It may indeed be fine. This is the central falsifiable question, and the first experiment settles it. A null result would itself be useful, because it would license vocabulary adaptation for multi-hop pipelines. We expect otherwise because the AgentVocab Telecom drop and the first-subword output initialization both point toward fidelity loss in generated entities that single-step multiple-choice tests cannot detect.
Why not just train the new embeddings longer? Longer tuning might separate sibling entities on its own. We will run the frequency-based baseline with four times the tuning budget. If the extra tuning closes the gap, the cheaper fix wins and our rule becomes unnecessary.
Decisive Checks
The first check is diagnostic. If frequency-based adaptation leaves hop-level bridge fidelity and Exact Match on MedHopQA within noise of the unadapted model, the motivating mechanism is wrong and the project reduces to a positive efficiency report. The second check is causal. Among bridge errors introduced by adaptation, the sibling-confusion rate should be markedly higher for merged entities than for unmerged ones. Errors concentrated on merged entities but not on prefix siblings would point to the compute-steps effect rather than initialization. The third check is the payoff. Role-aware merging should sit on or above the accuracy–attention-FLOPs frontier traced by the top-k sweep and by frequency-based merging, on both MedHopQA and MuSiQue, with total trajectory attention cost near the $(1-r)^2$ prediction. The fourth check is the control. The random-exclusion ablation should not reproduce the gain, and the reverse rule should do worst.
References
- AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation
- AgentVocab: Structure-Aware Vocabulary Adaptation for Efficient LLM Agents
- A Partition Cover Approach to Tokenization
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- DeepRAG: Integrating Hierarchical Reasoning and Process Supervision for Biomedical Multi-Hop QA
- Evaluating progress in automatic chest X-ray radiology report generation
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
- Squeezed Attention: Accelerating Long Context Length LLM Inference