TL;DR
This paper introduces Vocabulary Transfer (VT), a novel method to compress large language models (LLMs) by adapting their tokenizers to specific domains. By retraining tokenizers on in-domain data, VT significantly reduces model size and inference time with minimal performance degradation, offering a practical approach to deploying LLMs in real-world applications.
What Problem Does This Paper Address?
The rapid advancement of Natural Language Processing (NLP) has led to the development of increasingly large and powerful Language Models (LLMs) like GPT-3 and BLOOM. While these models achieve impressive performance, their sheer size presents significant challenges for real-world deployment. The computational cost for training, inference, and maintenance becomes prohibitive, especially for smaller businesses or applications with strict latency requirements. Existing compression techniques, such as Knowledge Distillation (KD), can reduce model size but often constrain the student model to use the same vocabulary as the larger teacher model. This paper addresses the gap of efficiently compressing LLMs by focusing on the often-overlooked component: the tokenizer and its associated vocabulary.
Methods
The core contribution of this paper is the Vocabulary Transfer (VT) technique. The intuition behind VT is that language varies significantly across different domains, and an "in-domain" tokenizer, trained on domain-specific data, can represent text more compactly. This leads to shorter token sequences, which in turn reduces computational complexity, as attention mechanisms in LLMs scale quadratically with sequence length. Furthermore, a smaller vocabulary directly translates to a smaller embedding matrix, further reducing model size.
The authors propose a simplified implementation called Fast Vocabulary Transfer (FVT). FVT initializes the embeddings of a new, in-domain vocabulary by leveraging the pre-trained embeddings of a general-purpose vocabulary. Specifically, if a token exists in both the general and in-domain vocabularies, its pre-trained embedding is directly used. For new tokens in the in-domain vocabulary, FVT averages the embeddings of the tokens produced by the general tokenizer that form the new token. This approach is applied after retraining the tokenizer on in-domain data.
VT is presented as a complementary technique that can be combined with other compression methods, notably Knowledge Distillation (KD). The paper explores combining VT with KD, where the student model is first distilled on a general domain and then on the in-domain data, followed by VT.
Datasets
The experiments were conducted on three distinct datasets representing different domains:
- ADE (Adverse Drug Events): A medical domain dataset for binary sentence classification. This domain is characterized by specialized technical terms, making it ideal for demonstrating the benefits of domain-specific tokenization.
- LEDGAR (Legal Document Classification): A legal domain dataset from contracts, annotated with 100 mutually-exclusive labels. This dataset is part of the LexGLUE benchmark.
- CoNLL03: A popular Named Entity Recognition (NER) benchmark using news stories. This dataset was chosen because its language is generally more standard, allowing for a comparison of VT's effectiveness in less specialized domains.
Key Insights and Ideas
- Vocabulary is a Compression Lever: The paper highlights that the tokenizer and its vocabulary are significant factors contributing to a model's inference speed and memory footprint, often overlooked in compression strategies.
- Domain-Specific Tokenization Pays Off: Adapting tokenizers to specific domains leads to more compact representations, reducing sequence lengths and computational costs.
- VT is Orthogonal to Other Methods: Vocabulary Transfer can be effectively combined with existing compression techniques like Knowledge Distillation, leading to synergistic improvements.
- Vocabulary Size is a Hyperparameter: The optimal vocabulary size for an in-domain tokenizer is not always the largest. Reducing vocabulary size can sometimes lead to better performance, suggesting a trade-off that needs careful consideration based on application KPIs.
- FVT Outperforms Simpler Initialization: The proposed FVT method, which uses a more sophisticated embedding initialization for new tokens, consistently outperforms a simpler Partial Vocabulary Transfer (PVT) baseline.
Key Results
- Significant Compression: VT alone achieved a remarkable 15%+ reduction in learnable parameters with almost no loss in F1 score.
- Speedup: VT enabled inference speedups ranging from 1.07x to 1.40x, with the most substantial gains observed in the specialized medical domain (ADE).
- Combined VT and KD: Combining VT with KD resulted in an overall reduction of up to 55% in model size and a speedup of up to 2.75x.
- Performance Trade-offs: While VT generally maintained performance, some specialized domains (like ADE) saw slight improvements in F1 score even with significant vocabulary reduction. In less specialized domains (CoNLL03), speedup diminished with smaller vocabularies.
- Vocabulary Size Impact: The paper demonstrated that reducing vocabulary size to 75%, 50%, or 25% of the original size had varying impacts, with some intermediate sizes yielding better results than full reduction.
Limitations and Future Work
- Limitations: The paper notes that in less specialized domains like news (CoNLL03), the benefits of domain-specific tokenization are less pronounced, and speedup can diminish with very small vocabularies. The authors also acknowledge that the optimal vocabulary size is a hyperparameter that needs careful tuning.
- Future Work: The authors plan to fully integrate Vocabulary Transfer within the Knowledge Distillation process to maximize information transfer. They also aim to define a unified metric that combines all KPIs (compression rate, inference speed, and accuracy) to facilitate model selection.
Summary updated