On the Effect of (Near) Duplicate Subwords in Language Modelling

Anton Schäfer, Thomas Hofmann, Imanol Schlag, Tiago Pimentel

Abstract

Modern language models (LMs) utilize subword tokenization, which frequently results in "near duplication"—treating semantically identical words (e.g., "apple" vs. "Apple") as distinct tokens. This study quantifies the impact of this redundancy, finding a "duplication tax" that reduces training efficiency by approximately 17%. While models naturally learn to align the embeddings of these duplicates over time, merging natural duplicates often harms performance, as they encode subtle but useful contextual information.

Introduction

Subword tokenization algorithms like Byte Pair Encoding (BPE) are designed for lossless text representation. However, they often create separate vocabulary entries for the same word based on capitalization, leading whitespace, or pluralization. This creates a potential efficiency gap: if a model learns about now, it does not inherently know that the same information applies to Now. This paper investigates whether this "near duplication" hinders learning and if deduplicating the vocabulary is a viable solution.

Methodology

The researchers employed two experimental frameworks using the Languini Kitchen environment and GPT-2 style transformers (85M to 370M parameters) trained on the books3 subset of The Pile.

  1. Synthetic Duplication: To establish an upper bound on the "duplication tax," the vocabulary was synthetically doubled. Every subword was given a "twin" token, and the model randomly encountered either version during training.
  2. Natural Deduplication: Real-world near duplicates were identified based on whitespace ($S_{space}$), capitalization ($S_{lower}$), and plurals ($S_{plural}$). Models were then trained with these groups merged into "canonical" tokens.

Performance was evaluated using projected perplexity ($PPL_{\mathbb{S}}$), a metric that accounts for vocabulary size differences by summing probabilities across equivalent token sets.

Results and Analysis

The 17% Efficiency Tax

Experiments with synthetic duplication revealed a significant performance penalty. Models with redundant vocabularies require more data to reach the same perplexity as those with clean vocabularies. Specifically, the "duplication tax" is estimated at roughly 17%, meaning a model with a clean vocabulary is 17% more data-efficient.

Perplexity comparison and training token efficiency between canonical and duplicated models.

Automatic Embedding Alignment

Despite the lack of explicit character-level information, LMs are surprisingly adept at discovering relationships between duplicates. As training progresses, the model aligns the vector representations (embeddings) of duplicate tokens based on their shared contexts. Cosine similarity between duplicates approaches 0.8 for frequently occurring tokens, indicating the model has learned their semantic equivalence.

Cosine similarity between embeddings of duplicate tokens as a function of training frequency.

Generalization and Local Context

The study also found that models generalize well between duplicates; a model fine-tuned on one version of a token (e.g., w) performs nearly as well on its twin (w'). However, merging natural duplicates (like now and Now) actually degraded performance, equivalent to losing 5-10% of training data. This suggests that "near" duplicates are not truly identical; capitalization and whitespace provide structural cues that help the model predict subsequent tokens. Furthermore, the presence of many duplicates within a short context window (the last 16 tokens) significantly increases model uncertainty.

Conclusion and Future Work

Near duplicates constitute a significant portion (35-46%) of modern LLM vocabularies. While they impose a training efficiency tax, they also carry nuanced contextual information that simple rule-based merging destroys. Future research should explore more sophisticated, learned deduplication methods that preserve structural signals while reducing redundancy. Additionally, investigating these effects in morphologically rich languages or at much larger model scales remains an open area for study.

Summary updated

Abstract

Tokenisation is a core part of language models (LMs). It involves splitting a character sequence into subwords which are assigned random indices before being served to the LM. However, this process—while typically lossless—may lead to less efficient LM training, because it removes character-level information, thereby making it more difficult to generalise across similar subwords, such as *now* and *Now*. We refer to such subwords as **near duplicates**. In this paper, we study the impact of near duplicate subwords on LM training efficiency. First, we design an experiment that gives us an upper bound to how much we should expect a model to improve if we could perfectly generalise across near duplicates. We do this, by duplicating each token in our LM’s vocabulary, creating perfectly equivalent classes of subwords. Experimentally, we find that LMs need roughly 17% more data when trained in a fully duplicated setting. Second, we investigate the impact of naturally occurring near duplicates on LMs. Here, we see that deduplicating them considerably hurts LM performance; but that this loss in performance can be easily mitigated.
BibTeX
@inproceedings{schafer2024effect,
  author = {Anton Schäfer and Thomas Hofmann and Imanol Schlag and Tiago Pimentel},
  title = {{On the Effect of (Near) Duplicate Subwords in Language Modelling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.571},
}
Figure