TL;DR
Large Language Models (LLMs) are widely deployed as chatbots, yet their tokenizers are traditionally optimized for general web text rather than conversational dialogue. This paper demonstrates that retraining tokenizers specifically on chatbot conversations can reduce token counts by 5% to over 10% during inference. This simple adjustment offers a direct path to meaningful computational and energy savings in real-world applications.
What Problem Does This Paper Address?
The computational and energy footprint of LLMs is a growing sustainability concern. Because the operational cost of LLM inference is directly tied to the number of tokens processed and generated, tokenization efficiency is critical.
Currently, tokenizers are trained on massive, general-purpose corpora (like web scrapes and books) to minimize token counts across broad text types. However, a massive portion of real-world LLM usage is conversational. The authors address the mismatch between what tokenizers are optimized for (general text) and how LLMs are actually used (conversations), asking whether domain-specific tokenizers can unlock significant efficiency gains.
Methods
The authors evaluate a representative suite of tokenizers from popular open and proprietary LLMs, including GPT-4, LLaMA-3.1, Gemma-2, Mistral-7B, BLOOM, DeepSeek-R1, and Phi-4. They measure "fertility"—the average number of tokens per word—to quantify tokenization efficiency.
To build conversation-optimized tokenizers, they retrain the original tokenization algorithms (such as Byte Pair Encoding) using three distinct strategies based on conversational data:
- Optimizing using only user inputs.
- Optimizing using only chatbot responses.
- Optimizing using full conversations (both inputs and responses).
They then compare these new tokenizers against the originals on both conversational test data and standard pre-training text.
Datasets
- LMSYS Chat 1M: A dataset containing one million real-world conversations between anonymous users and 25 different LLMs in multiple languages. This was used for retraining (80% split) and evaluating (20% split) the conversational tokenizers.
- Colossal Clean Crawled Corpus (C4): A cleaned version of Common Crawl developed by AllenAI, representing standard LLM pre-training data. A representative subset was used to evaluate how conversation-optimized tokenizers perform on general text.
Key Insights and Ideas
- Conversational Mismatch: Standard tokenizers consistently exhibit higher fertility (worse compression) on chatbot dialogues than on standard training text, confirming that current designs are sub-optimal for conversational deployments.
- The "Sub-optimal Original" Surprise: For three models (Gemma-2-9b, Mistral-7B, and BLOOM), optimizing the tokenizer for conversations also reduced token counts when applied back to the general C4 training corpus. This suggests that the original tokenizers for these models possess general inefficiencies independent of the conversational domain.
- Data Distribution Matters: While multilingual performance generally improved, DeepSeek-R1's performance on Chinese text actually degraded. This highlights a crucial caveat: if a specific language is underrepresented in the conversational tuning dataset (Chinese made up only 2.4% of LMSYS Chat 1M), domain optimization can inadvertently penalize it compared to the original training corpus.
Key Results
Retraining tokenizers on full conversations yielded the highest token reductions across the board. As shown in Figure 1, baseline fertility is uniformly higher for user inputs than chatbot outputs, though full conversation fertility tracks closely with outputs since they contain the majority of the words.
Token count reductions on conversational data ranged from ~5% for DeepSeek-R1, LLaMA-3.1-8B, and Phi-4, to over 10% for Gemma-2-9b, Mistral-7B, and BLOOM (reaching up to 13.29% for BLOOM), as detailed in Figure 2.
On the general C4 corpus (Figure 4), the conversation-optimized tokenizers for Gemma, Mistral, and BLOOM achieved unexpected token reductions of 1% to 5.5%, while the other models saw only negligible token increases (<2%).
Limitations and Future Work
- Dataset Scope: The study relies on a single conversational dataset (LMSYS Chat 1M) and one training corpus (C4). Additional datasets are needed to ensure generalizability.
- Inference vs. Training: The authors only evaluate the inference phase. A full lifecycle analysis must consider how modifying the tokenizer impacts the computationally expensive LLM pre-training phase.
- Downstream Performance: The paper does not evaluate whether changing the token vocabulary alters the actual quality, reasoning capabilities, or accuracy of the LLM's generations.
- Code and Language Mismatches: The impact on programming code tokenization and highly specific language distributions remains unknown and requires careful balancing to avoid penalizing low-resource languages.
Summary updated