Abstract
Large language model (LLM) inference is computation and memory intensive, so we adapt lexical shortlisting to it hoping to improve both. While lexical shortlisting is well-explored in tasks like machine translation, it requires modifications before being suitable for LLMs as the intended applications vary significantly. Our work studies two heuristics to shortlist sub-vocabulary at LLM inference time: Unicode-based script filtering and corpus-based selection. We explore different LLM families and sizes, and we find that lexical shortlisting can reduce the memory usage of some models by nearly 50% and has an upper bound of 25% improvement in generation speed. In this pilot study, we also identify the drawbacks of such vocabulary selection methods and propose avenues for future research.
Introduction
Modern Large Language Models (LLMs) are designed as massive "polyglots," supporting dozens of languages through expansive vocabularies. For instance, the BLOOM model family utilizes a vocabulary ($V$) of over 250,000 tokens. During inference, the model must perform a computationally expensive matrix multiplication ($H \times |V|$) at the final layer to calculate probabilities for every possible token. In many practical scenarios, the target language is known in advance. This research investigates "Vocabulary Trimming" (VT)—the process of removing irrelevant tokens from the output layer—to determine if inference can be made more efficient without sacrificing performance.
Methodology
We evaluated two primary heuristics for creating language-specific sub-vocabularies:
- Script-based Filtering: This method uses Unicode ranges to retain only tokens belonging to a specific writing system (e.g., Cyrillic for Bulgarian). While interpretable, it is rigid and may exclude necessary characters like punctuation or numbers.
- Corpus-based Selection: We tokenized a representative corpus (WikiMatrix) for each target language and retained only the tokens present in that text. This approach is more flexible, naturally including common symbols and cross-lingual tokens.
Additionally, an Oracle experiment was conducted to establish a theoretical upper bound by retaining only the tokens the model actually used in its final output.
Memory Efficiency
Vocabulary trimming has a profound impact on the model's memory footprint, particularly for smaller architectures where the embedding and output layers constitute a significant portion of the total parameters. For the BLOOM-560M model, the vocabulary embeddings account for nearly 50% of the total parameters. By applying VT, the memory required for these layers can be reduced by up to 90%.
As shown in the data, for BLOOM-560M, the full vocabulary layer requires 0.90 GB, which can be trimmed to as little as 0.09 GB for Bulgarian Unicode. However, as models scale to 7B parameters and beyond, the hidden layer size ($H$) increases while the vocabulary size often remains static, leading to diminishing returns in overall memory savings.
Quality and Inference Performance
The effectiveness of VT varies significantly by hardware and language. On CPU-based inference, smaller models saw generation speedups of up to 25%. However, for 7B parameter models, these gains diminished to approximately 5–10%.
Regarding output quality, Latin-based languages like English and Spanish maintained nearly identical performance to the full models. For non-Latin languages, the corpus-based method proved superior to script-based filtering. The latter often suffered from the "Code-Mixing Trap," where the aggressive removal of Latin characters prevented the model from generating necessary technical terms, URLs, or common English loanwords, resulting in a higher number of "misses" and lower BLEU scores.
The GPU Paradox and Limitations
A critical finding of this study is that vocabulary trimming provides virtually zero speedup on GPUs. Modern GPU architectures are highly optimized for large matrix multiplications; the overhead introduced by slicing matrices into smaller, non-standard shapes often outweighs the computational savings.
Furthermore, static trimming lacks the flexibility to handle dynamic language shifts within a single session. If a user switches languages mid-conversation, a trimmed model will fail to produce the correct tokens. Future research should focus on dynamic vocabulary adjustment and more sophisticated handling of code-mixed environments to make VT viable for production-grade, GPU-accelerated LLM inference.
Summary updated