Abstract
Modern language models (LMs) utilize subword tokenization, which frequently results in "near duplication"—treating semantically identical words (e.g., "apple" vs. "Apple") as distinct tokens. This study quantifies the impact of this redundancy, finding a "duplication tax" that reduces training efficiency by approximately 17%. While models naturally learn to align the embeddings of these duplicates over time, merging natural duplicates often harms performance, as they encode subtle but useful contextual information.
Introduction
Subword tokenization algorithms like Byte Pair Encoding (BPE) are designed for lossless text representation. However, they often create separate vocabulary entries for the same word based on capitalization, leading whitespace, or pluralization. This creates a potential efficiency gap: if a model learns about now, it does not inherently know that the same information applies to Now. This paper investigates whether this "near duplication" hinders learning and if deduplicating the vocabulary is a viable solution.
Methodology
The researchers employed two experimental frameworks using the Languini Kitchen environment and GPT-2 style transformers (85M to 370M parameters) trained on the books3 subset of The Pile.
- Synthetic Duplication: To establish an upper bound on the "duplication tax," the vocabulary was synthetically doubled. Every subword was given a "twin" token, and the model randomly encountered either version during training.
- Natural Deduplication: Real-world near duplicates were identified based on whitespace ($S_{space}$), capitalization ($S_{lower}$), and plurals ($S_{plural}$). Models were then trained with these groups merged into "canonical" tokens.
Performance was evaluated using projected perplexity ($PPL_{\mathbb{S}}$), a metric that accounts for vocabulary size differences by summing probabilities across equivalent token sets.
Results and Analysis
The 17% Efficiency Tax
Experiments with synthetic duplication revealed a significant performance penalty. Models with redundant vocabularies require more data to reach the same perplexity as those with clean vocabularies. Specifically, the "duplication tax" is estimated at roughly 17%, meaning a model with a clean vocabulary is 17% more data-efficient.
Automatic Embedding Alignment
Despite the lack of explicit character-level information, LMs are surprisingly adept at discovering relationships between duplicates. As training progresses, the model aligns the vector representations (embeddings) of duplicate tokens based on their shared contexts. Cosine similarity between duplicates approaches 0.8 for frequently occurring tokens, indicating the model has learned their semantic equivalence.
Generalization and Local Context
The study also found that models generalize well between duplicates; a model fine-tuned on one version of a token (e.g., w) performs nearly as well on its twin (w'). However, merging natural duplicates (like now and Now) actually degraded performance, equivalent to losing 5-10% of training data. This suggests that "near" duplicates are not truly identical; capitalization and whitespace provide structural cues that help the model predict subsequent tokens. Furthermore, the presence of many duplicates within a short context window (the last 16 tokens) significantly increases model uncertainty.
Conclusion and Future Work
Near duplicates constitute a significant portion (35-46%) of modern LLM vocabularies. While they impose a training efficiency tax, they also carry nuanced contextual information that simple rule-based merging destroys. Future research should explore more sophisticated, learned deduplication methods that preserve structural signals while reducing redundancy. Additionally, investigating these effects in morphologically rich languages or at much larger model scales remains an open area for study.
Summary updated