Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMs

Chengyuan Liu, Shihang Wang, Lizhi Qing, Kun Kuang, Yangyang Kang, Changlong Sun, Fei Wu

TL;DR

When adapting Large Language Models (LLMs) to specialized fields like law or medicine, bigger isn't always better. This paper introduces VEGAD, a gradient-based method that "pans for gold" in a candidate vocabulary to identify the most impactful words for a specific domain. By selecting an optimal subset rather than the entire dictionary, VEGAD boosts domain performance while significantly mitigating "catastrophic forgetting" of general reasoning and math skills.

What Problem Does This Paper Address?

Standard LLMs often struggle with domain-specific jargon and concepts. To fix this, researchers usually expand the model's vocabulary before fine-tuning. The logic is simple: if the model recognizes "hyperthyroidism" as a single token instead of five separate sub-words, it becomes more efficient and understands the concept better.

However, the authors identify a major pitfall: indiscriminate expansion. Most current methods use frequency-based tools (like SentencePiece) or expert-curated lists to add thousands of new words. The authors' pilot study reveals that adding too many words actually degrades the model’s performance on general tasks (like math and safety) and can even hurt the very domain performance it was meant to help. The challenge is identifying which words are truly "valuable" to the model's learning process and which are just noise.

Methods

The authors propose VEGAD (Vocabulary Expansion via GrADients). Instead of looking at how often a word appears, VEGAD looks at how much a word matters to the model's loss function.

The VEGAD pipeline for gradient-based vocabulary expansion.
  1. Trie-Based Candidate Tracking: They first build a "Trie" (a prefix tree) of all potential candidate words from a domain corpus.
  2. Gradient Calculation: As the model processes domain data, VEGAD tracks the gradients of the tokens. Crucially, they calculate gradients for both the embedding layer (input) and the Language Modeling (LM) head (output). Most previous studies only looked at the embeddings, but this paper argues the output head is vital for generation tasks.
  3. Gold Panning: Words that consistently show high gradients are deemed "crucial" for the task. The top $K$ words with the highest cumulative gradients are selected for the final expanded vocabulary.
  4. Efficiency Optimizations: To make this computationally feasible, they use the Aho–Corasick algorithm and prefix accumulation, allowing them to calculate gradients for thousands of candidate words simultaneously without a massive slowdown.

Key Insights and Ideas

  • The "Optimal Subset" Discovery: The most striking finding is that model performance follows a "peak" curve. Adding a few thousand well-chosen words helps, but adding the full set of available domain words causes a sharp decline in general reasoning (GSM8K scores dropped by over 70% in some baseline tests).
Performance trends across different vocabulary sizes, highlighting the 'optimal subset' effect.
  • Gradients > Frequency: A word might appear frequently but provide little "new" information to the model. High gradients signal that the model is actively "learning" or struggling with that specific term, making it a better candidate for a dedicated token.
  • The LM Head Matters: Ablation studies showed that ignoring the gradients in the LM head (the layer that predicts the next token) leads to a significant drop in the quality of generated text.
  • Mitigating Catastrophic Forgetting: By keeping the vocabulary expansion surgical and precise, VEGAD retains more of the model's original "pre-trained" knowledge compared to frequency-based methods like Jieba or SentencePiece.

Key Results

VEGAD consistently outperformed baselines (including direct fine-tuning and expert-designed vocabularies like LawGPT) across the board:

  • Legal Domain: On the Article QA dataset, VEGAD achieved a BLEU score of 28.58, surpassing the "Jieba" baseline (28.04) and the standard SFT (26.70).
  • Medical Domain: In the CMDD dataset, VEGAD reached a BLEU of 5.84, a significant relative improvement over the base model.
  • General Reasoning: While all fine-tuning causes some "forgetting," VEGAD was much gentler. On the GSM8K math benchmark, while other expansion methods saw accuracy plummet to ~6-9%, VEGAD maintained a much higher 15.20% (closer to the original model's 22.10%).

Limitations and Future Work

The authors acknowledge that while they've solved the selection problem, the initialization problem remains. When you add a new word, you have to give it a starting "weight" (vector). Currently, they use simple averaging of sub-token weights. They found that more complex initialization methods (like using Wikipedia data) didn't consistently help, suggesting that finding the "perfect" starting point for a new token is still an open research question.

Additionally, the method was primarily tested on Chinese datasets; while they showed a small successful experiment on English (PubMedQA), broader cross-lingual validation is a logical next step.

Summary updated

Abstract

While Large Language Models (LLMs) demonstrate impressive generation abilities, they frequently struggle when it comes to specialized domains due to their limited domain-specific knowledge. Studies on domain-specific LLMs resort to expanding the vocabulary before fine-tuning on domain-specific corpus, aiming to decrease the sequence length and enhance efficiency during decoding, without thoroughly investigating the results of vocabulary expansion to LLMs over different domains. Our pilot study reveals that expansion with only a subset of the entire vocabulary may lead to superior performance. Guided by the discovery, this paper explores how to identify a vocabulary subset to achieve the optimal results. We introduce VEGAD, an adaptive method that automatically identifies valuable words from a given domain vocabulary. Our method has been validated through experiments on three Chinese datasets, demonstrating its effectiveness. Additionally, we have undertaken comprehensive analyses of the method. The selection of a optimal subset for expansion has shown to enhance performance on both domain-specific tasks and general tasks, showcasing the potential of VEGAD.
BibTeX
@inproceedings{liu2024gold,
  author = {Chengyuan Liu and Shihang Wang and Lizhi Qing and Kun Kuang and Yangyang Kang and Changlong Sun and Fei Wu},
  title = {{Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.424},
}
Figure