Is There a Case for Conversation Optimized Tokenizers in Large Language Models?

Raquel Ferrando, Javier Conde, Gonzalo Martínez, Pedro ReviriegoarXiv 2025 · arXiv

TL;DR

Large Language Models (LLMs) are widely deployed as chatbots, yet their tokenizers are traditionally optimized for general web text rather than conversational dialogue. This paper demonstrates that retraining tokenizers specifically on chatbot conversations can reduce token counts by 5% to over 10% during inference. This simple adjustment offers a direct path to meaningful computational and energy savings in real-world applications.

What Problem Does This Paper Address?

The computational and energy footprint of LLMs is a growing sustainability concern. Because the operational cost of LLM inference is directly tied to the number of tokens processed and generated, tokenization efficiency is critical.

Currently, tokenizers are trained on massive, general-purpose corpora (like web scrapes and books) to minimize token counts across broad text types. However, a massive portion of real-world LLM usage is conversational. The authors address the mismatch between what tokenizers are optimized for (general text) and how LLMs are actually used (conversations), asking whether domain-specific tokenizers can unlock significant efficiency gains.

Methods

The authors evaluate a representative suite of tokenizers from popular open and proprietary LLMs, including GPT-4, LLaMA-3.1, Gemma-2, Mistral-7B, BLOOM, DeepSeek-R1, and Phi-4. They measure "fertility"—the average number of tokens per word—to quantify tokenization efficiency.

To build conversation-optimized tokenizers, they retrain the original tokenization algorithms (such as Byte Pair Encoding) using three distinct strategies based on conversational data:

  1. Optimizing using only user inputs.
  2. Optimizing using only chatbot responses.
  3. Optimizing using full conversations (both inputs and responses).

They then compare these new tokenizers against the originals on both conversational test data and standard pre-training text.

Datasets

  • LMSYS Chat 1M: A dataset containing one million real-world conversations between anonymous users and 25 different LLMs in multiple languages. This was used for retraining (80% split) and evaluating (20% split) the conversational tokenizers.
  • Colossal Clean Crawled Corpus (C4): A cleaned version of Common Crawl developed by AllenAI, representing standard LLM pre-training data. A representative subset was used to evaluate how conversation-optimized tokenizers perform on general text.

Key Insights and Ideas

  • Conversational Mismatch: Standard tokenizers consistently exhibit higher fertility (worse compression) on chatbot dialogues than on standard training text, confirming that current designs are sub-optimal for conversational deployments.
  • The "Sub-optimal Original" Surprise: For three models (Gemma-2-9b, Mistral-7B, and BLOOM), optimizing the tokenizer for conversations also reduced token counts when applied back to the general C4 training corpus. This suggests that the original tokenizers for these models possess general inefficiencies independent of the conversational domain.
  • Data Distribution Matters: While multilingual performance generally improved, DeepSeek-R1's performance on Chinese text actually degraded. This highlights a crucial caveat: if a specific language is underrepresented in the conversational tuning dataset (Chinese made up only 2.4% of LMSYS Chat 1M), domain optimization can inadvertently penalize it compared to the original training corpus.

Key Results

Retraining tokenizers on full conversations yielded the highest token reductions across the board. As shown in Figure 1, baseline fertility is uniformly higher for user inputs than chatbot outputs, though full conversation fertility tracks closely with outputs since they contain the majority of the words.

Figure 1: Fertility on the LLM training dataset (C4) and on the conversational dataset (LMSYS). For the conversa- tional dataset fertility is computed on the input (user), output (assistant) and entire conversations.

Token count reductions on conversational data ranged from ~5% for DeepSeek-R1, LLaMA-3.1-8B, and Phi-4, to over 10% for Gemma-2-9b, Mistral-7B, and BLOOM (reaching up to 13.29% for BLOOM), as detailed in Figure 2.

Figure 2: Reduction on the number of tokens for the conversation-optimized tokenizers on the conversational test set (LMSYS).

On the general C4 corpus (Figure 4), the conversation-optimized tokenizers for Gemma, Mistral, and BLOOM achieved unexpected token reductions of 1% to 5.5%, while the other models saw only negligible token increases (<2%).

Figure 4: Increase on the number of tokens for the conversation-optimized tokenizers on the LLM training corpus (C4).

Limitations and Future Work

  • Dataset Scope: The study relies on a single conversational dataset (LMSYS Chat 1M) and one training corpus (C4). Additional datasets are needed to ensure generalizability.
  • Inference vs. Training: The authors only evaluate the inference phase. A full lifecycle analysis must consider how modifying the tokenizer impacts the computationally expensive LLM pre-training phase.
  • Downstream Performance: The paper does not evaluate whether changing the token vocabulary alters the actual quality, reasoning capabilities, or accuracy of the LLM's generations.
  • Code and Language Mismatches: The impact on programming code tokenization and highly specific language distributions remains unknown and requires careful balancing to avoid penalizing low-resource languages.

Summary updated

Abstract

The computational and energy costs of Large Language Models (LLMs) have increased exponentially driven by the growing model sizes and the massive adoption of LLMs by hundreds of millions of users. The unit cost of an LLM is the computation of a token. Therefore, the tokenizer plays an important role in the efficiency of a model, and they are carefully optimized to minimize the number of tokens for the text in their training corpus. One of the most popular applications of LLMs are chatbots that interact with users. A key observation is that, for those chatbots, what is important is the performance of the tokenizer in the user text input and the chatbot responses. Those are most likely different from the text in the training corpus. So, a question that immediately arises is whether there is a potential benefit in optimizing tokenizers for chatbot conversations. In this paper, this idea is explored for different tokenizers by using a publicly available corpus of chatbot conversations to redesign their vocabularies and evaluate their performance in this domain. The results show that conversation-optimized tokenizers consistently reduce the number of tokens in chatbot dialogues, which can lead to meaningful energy savings, in the range of 5% to 10% while having minimal or even slightly positive impact on tokenization efficiency for the original training corpus.
BibTeX
@misc{ferrando2025is,
  author = {Raquel Ferrando and Javier Conde and Gonzalo Martínez and Pedro Reviriego},
  title = {{Is There a Case for Conversation Optimized Tokenizers in Large Language Models?}},
  year = {2025},
  eprint = {2506.18674},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.18674},
}
Figure