Abstract
While Large Language Models (LLMs) demonstrate impressive generation abilities, they frequently struggle when it comes to specialized domains due to their limited domain-specific knowledge. Studies on domain-specific LLMs resort to expanding the vocabulary before fine-tuning on domain-specific corpus, aiming to decrease the sequence length and enhance efficiency during decoding, without thoroughly investigating the results of vocabulary expansion to LLMs over different domains. Our pilot study reveals that expansion with only a subset of the entire vocabulary may lead to superior performance. Guided by the discovery, this paper explores how to identify a vocabulary subset to achieve the optimal results. We introduce VEGAD, an adaptive method that automatically identifies valuable words from a given domain vocabulary. Our method has been validated through experiments on three Chinese datasets, demonstrating its effectiveness. Additionally, we have undertaken comprehensive analyses of the method. The selection of a optimal subset for expansion has shown to enhance performance on both domain-specific tasks and general tasks, showcasing the potential of VEGAD.
Introduction
When LLMs encounter specialized domains like law or medicine, their base vocabulary—typically trained on general web text—often fails to capture specialized terminology. Standard tokenizers decompose these complex terms into multiple sub-word "shards," which decreases decoding efficiency and can degrade semantic understanding.
While vocabulary expansion is a common strategy to address this, research indicates that simply adding more tokens is not always beneficial. A pilot study conducted by the authors revealed a critical performance peak: adding too many words can actually degrade model performance. This necessitates a "gold panning" approach—identifying the specific vocabulary "nuggets" that provide maximum benefit without bloating the model or causing catastrophic forgetting of general knowledge.
Methodology: The VEGAD Framework
To solve the selection problem, the authors introduce VEGAD (Vocabulary Expansion via GrADients), an adaptive method that treats vocabulary selection as a gradient-driven optimization problem. The framework follows a structured pipeline to identify the most impactful tokens for a target domain.
The VEGAD Process:
- Candidate Generation: A large pool of potential domain words is extracted using tools like Jieba or expert-curated lists.
- Trie Construction: These candidates are organized into a prefix tree (Trie) for efficient searching within the domain corpus.
- Gradient Calculation: As the model processes domain data, VEGAD calculates gradients for both the embedding layer and the Language Modeling (LM) head. Including the LM head gradients is a key innovation, as it directly impacts the model's generation quality.
- Selection: Only the top $K$ words with the highest cumulative gradients—representing the tokens the model "struggles" with most or finds most informative—are selected.
- Efficiency Optimization: To handle large-scale datasets, the method utilizes the Aho-Corasick algorithm and prefix accumulation to calculate these gradients in a single pass.
- Fine-tuning: The model is resized to include the new tokens and then undergoes Supervised Fine-Tuning (SFT).
Key Results and Insights
VEGAD was evaluated across legal and medical benchmarks, as well as general capability tests like ALPACA (instruction following) and GSM8K (mathematical reasoning).
- Domain Superiority: In the legal domain (Article QA), VEGAD achieved a 28.58 BLEU score, significantly outperforming the standard SFT baseline (26.70). In the medical domain (CMDD), it showed a +11.45% relative improvement over the general LLM.
- Mitigating Catastrophic Forgetting: Vocabulary expansion acts as a buffer. By adding new parameters for domain concepts, the model avoids aggressively overwriting its existing general-purpose embeddings. For instance, while standard fine-tuning caused math accuracy (GSM8K) to drop to 6.60%, VEGAD-expanded models maintained 15.20%.
- The Importance of the LM Head: The research demonstrates that the output layer (LM Head) is as critical as the input layer (Embeddings) for expansion; ignoring it leads to a noticeable drop in generation performance.
Conclusion
VEGAD offers a robust, adaptive solution for domain-specific LLM adaptation. By selecting a high-gradient subset of candidate words rather than indiscriminately expanding the lexicon, it improves domain performance while preserving general reasoning skills. Future work will focus on improving the weight initialization of new tokens and exploring the dynamics of vocabulary expansion in larger model scales (70B+).
Summary updated