On the Effect of (Near) Duplicate Subwords in Language Modelling

Anton Schäfer, Thomas Hofmann, Imanol Schlag, Tiago Pimentel

Abstract

Modern language models are "taxed" by their own vocabularies: nearly 40% of the subwords they learn are near-duplicates (such as "now" vs. "Now"). This research reveals that while models can learn to treat these as semantically equivalent, doing so costs them roughly 10% in data efficiency. Surprisingly, simply merging these duplicates often hurts performance because they carry subtle, unique contextual information that aids in next-token prediction.

Introduction

Most Large Language Models (LLMs) do not process raw text; instead, they operate on tokens—subword units like "ing" or " apple". Due to the mechanics of subword tokenization (such as Byte Pair Encoding), vocabularies often contain many "near-duplicates": pairs that are semantically almost identical but differ by capitalization, leading/trailing whitespace, or pluralization (e.g., _the vs the). This study investigates the "Efficiency Gap"—the training data wasted by learning the same concept multiple times—and the "Semantic Trap"—the potential loss of information when these tokens are merged.

Methodology

The researchers utilized the Languini Kitchen framework to conduct a two-pronged experimental strategy:

  1. Perfect Duplication (The Upper Bound): To measure the data tax of redundancy, the authors synthetically doubled a model's vocabulary. For every token, a "twin" was created with the exact same meaning. During training, the model randomly encountered either the original or the twin, creating a controlled environment to measure generalization.
  2. Natural Deduplication: The authors merged existing LLM vocabulary tokens based on four heuristics: S_space (ignoring whitespace), S_lower (ignoring capitalization), S_plural (ignoring plural suffixes), and S_all (combining all three).
  3. Projected Perplexity: To ensure fair comparisons across models with different vocabulary sizes, a new metric was developed. This metric sums the probabilities of all "equivalent" tokens, ensuring a model is not penalized simply for having more choices.

Experimental Setup

Experiments were conducted using the Languini training data (a filtered version of the books3 subset from The Pile). Small models were trained on 1.2B tokens, while medium models reached 2.8B tokens. The base vocabulary consisted of 16k BPE tokens, which was then expanded or contracted for specific tests.

The Efficiency Tax

The study found that redundancy comes with a measurable cost. In the synthetic "perfect duplicate" environment, a model with a 100% duplicated vocabulary requires approximately 17% more data to reach the same performance level as a non-redundant model. For a typical LLM, which contains roughly 40% near-duplicates, this translates to a 10% loss in data efficiency.

The relationship between duplication and performance degradation appears to be roughly linear. Doubling 70% of the vocabulary—a rate similar to many modern LLMs—leads to a 10% decrease in data efficiency.

The Semantic Value of Redundancy

Perhaps the most counter-intuitive finding is that merging real-world near-duplicates (like Now and now) actually decreases performance. When the authors applied the S_all merging heuristic, the resulting performance degradation was equivalent to losing 5-10% of the training data.

This suggests that "near-duplicates" are not truly redundant. Instead, they encode vital information about sentence structure, emphasis, and local context. For example, a capitalized word at the start of a sentence provides different structural cues than its lowercase counterpart in the middle of a phrase. While character-level models avoid this subword redundancy, they may struggle because they lose these specific "hints" provided by distinct subword forms.

Embedding Alignment and Generalization

The model's ability to realize that two tokens share a meaning is driven primarily by frequency. In synthetic tests, duplicate pairs reached an average cosine similarity of 0.8. However, in natural settings, this dropped to 0.4, reinforcing the idea that near-duplicates are semantically distinct to the model.

Interestingly, the models demonstrated strong generalization capabilities. A model trained on duplicates but fine-tuned on only one version of a word (e.g., apple) generalized perfectly to the other version (Apple) during GLUE benchmark testing. This indicates that the underlying "circuits" learned by the model are indeed shared across redundant tokens, even if the embeddings themselves remain somewhat distinct.

Limitations and Future Directions

This study focused on English-language models up to 370M parameters. It remains to be seen if massive models (70B+ parameters) are better at overcoming this redundancy or if they are even more sensitive to these subtle differences. Furthermore, languages with richer morphology (like Turkish or Finnish) may exhibit different duplication effects.

The authors suggest that "character-aware" subword embeddings—where the model can perceive the characters within a token—might eventually offer the best of both worlds: the efficiency of a deduplicated vocabulary without the loss of contextual information provided by specific subword forms.

Summary updated

BibTeX
@inproceedings{schafer2024effect,
  author = {Anton Schäfer and Thomas Hofmann and Imanol Schlag and Tiago Pimentel},
  title = {{On the Effect of (Near) Duplicate Subwords in Language Modelling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.571},
}
Figure