The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics

Nikolay Bogoychev, Pinzhen Chen, Barry Haddow, Alexandra Birch

TL;DR

This paper explores vocabulary trimming (VT) as a method to speed up and reduce memory usage for large language models (LLMs) during inference. They found that simple, language-aware heuristics can offer significant gains, especially for smaller models and Latin-based languages, but performance varies and diminishes with larger models.

What Problem Does This Paper Address?

Deploying large language models (LLMs) is computationally expensive, particularly during inference. A major bottleneck is the matrix multiplication in the output layer, which scales with the size of the model's vocabulary. This operation consumes significant memory and time, especially given the autoregressive nature of LLM decoding. The authors investigate "vocabulary trimming" (VT) – reducing the vocabulary size by only loading relevant embedding entries – as a way to mitigate these costs. While VT has been explored in machine translation, its application to LLMs is more challenging due to their open-ended and variable output, making it harder to determine the necessary vocabulary.

Methods

The paper proposes two straightforward, language-heuristic-based methods for vocabulary trimming:

  • Unicode-based script filtering: This method removes vocabulary items that do not belong to the target language's Unicode script. It's particularly effective for languages with unique scripts.
  • Corpus-based selection: This approach involves tokenizing a representative corpus in the desired language and selecting vocabulary entries that appear in it.

These methods are applied to LLMs from different families (BLOOM, LLaMA) and sizes. Crucially, the input tokens for a given batch are always included in the sub-vocabulary to ensure they can be processed. The authors also keep the top 300 vocabulary entries, which typically correspond to special tokens, to maintain essential functionality.

Datasets

The experiments use a set of 50 prompt questions from OpenAssistant (Köpf et al., 2023), which were then human-translated into the target languages. The models were evaluated on Bulgarian, Chinese, English, and Spanish. For corpus-based selection, a subset of WikiMatrix (Schwenk et al., 2021) was used.

Key Insights and Ideas

  • Language matters: The effectiveness of VT is highly dependent on the language. Simple script-based filtering works well for Latin-based languages but struggles with code-mixing or non-Latin scripts.
  • Model size is a factor: Smaller LLMs benefit more from VT, with nearly 50% memory reduction observed. As models grow larger, the proportion of parameters dedicated to embeddings decreases, diminishing the impact of VT.
  • Trade-offs exist: While VT can improve speed and reduce memory, it doesn't guarantee identical output compared to a full vocabulary. "Misses" (differences in output) are observed, especially for non-Latin languages.
  • Orthogonal to other methods: The authors suggest VT can be applied alongside other efficiency techniques like attention mechanisms or quantization.

Key Results

  • Memory Reduction: For smaller models (e.g., BLOOM-560M), VT reduced memory usage by nearly 50%.
  • Speed Improvement: An upper bound of 25% improvement in generation speed was observed. However, the actual speedup varied, with smaller BLOOM models showing around 20% improvement, while larger ones saw only 5-10%.
  • Language Performance:
    • Unicode filtering: Reduced Bulgarian and Chinese vocabularies to 10-20%, but only to 60% for English and Spanish. It performed well for Latin languages but harmed code-mixing scenarios.
  • Corpus-based selection: Led to smaller vocabulary reductions but was more balanced.
  • Misses: For English and Spanish, models with VT maintained almost identical outputs. However, Bulgarian and Chinese saw higher "miss" rates (10-20% and 30-40% respectively).
  • GPU Performance: On GPUs, the proposed VT methods did not lead to improved inference speed, likely because GPUs are optimized for large matrix operations, and the reduction in matrix size wasn't significant enough to overcome overheads.

Limitations and Future Work

The paper acknowledges several limitations:

  • Diminishing returns: The effectiveness of VT decreases with larger models.
  • Language-specific issues: Unicode filtering is less effective for languages requiring code-mixing or for English-centric LLMs with smaller vocabularies.
  • Quality trade-offs: VT can lead to output differences ("misses"), particularly for non-Latin languages.
  • Corpus-based limitations: This method might miss rare but valid tokens or suffer from domain mismatch.

Future work could explore more sophisticated heuristics for vocabulary selection, investigate how VT interacts with other efficiency techniques, and develop methods that better balance efficiency gains with output quality across a wider range of languages and model sizes.

Summary updated

Abstract

Large language model (LLM) inference is computation and memory intensive, so we adapt lexical shortlisting to it hoping to improve both. While lexical shortlisting is well-explored in tasks like machine translation, it requires modifications before being suitable for LLMs as the intended applications vary significantly. Our work studies two heuristics to shortlist sub-vocabulary at LLM inference time: Unicode-based script filtering and corpus-based selection. We explore different LLM families and sizes, and we find that lexical shortlisting can reduce the memory usage of some models by nearly 50\% and has an upper bound of 25\% improvement in generation speed. In this pilot study, we also identify the drawbacks of such vocabulary selection methods and propose avenues for future research.
BibTeX
@inproceedings{bogoychev2024ups,
  author = {Nikolay Bogoychev and Pinzhen Chen and Barry Haddow and Alexandra Birch},
  title = {{The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure