Which Pieces Does Unigram Tokenization Really Need?

Sander Land, Yuval Pinter

Abstract

Unigram tokenization is a probabilistic alternative to Byte-Pair Encoding (BPE) but is often viewed as a "black box" due to complex implementations like SentencePiece. This work systematically deconstructs the Unigram training pipeline—initialization, EM estimation, and pruning—to identify essential components. We demonstrate that simplified "Final-Style Pruning" (FSP) achieves BPE-level compression while maintaining superior morphological alignment. Our findings suggest that many standard heuristic complexities are unnecessary for high-performance tokenization.

Introduction

In Large Language Models (LLMs), tokenization serves as the critical interface between raw text and numerical representations. While Byte-Pair Encoding (BPE) is the industry standard due to its simplicity and greedy efficiency, Unigram tokenization offers a more principled probabilistic approach. However, Unigram's adoption is hindered by its implementation complexity; standard tools like SentencePiece contain numerous hard-coded parameters and opaque logic. This paper investigates which of these moving parts are truly necessary, aiming to simplify the algorithm without sacrificing performance.

Methods

The authors deconstruct the traditional three-stage Unigram training pipeline to isolate performance drivers:

  1. Seed Vocabulary Initialization: Standard methods use complex full-text suffix arrays. This study proposes Pretoken-based Initialization, starting with pre-split words, which proves more efficient.
  2. The EM Algorithm: The researchers tested the necessity of multiple EM iterations and complex mathematical transformations (like the "digamma" function).
  3. Iterative Pruning: They introduce Final-Style Pruning (FSP), a simplified heuristic that prunes tokens based on raw probabilities rather than the computationally expensive "second-best tokenization" likelihood used in standard implementations.

Key Insights and Ideas

The Compression-Objective Tradeoff

A central finding is the existence of a fundamental tradeoff in tokenization. Optimization can target either the best probabilistic model (lowest loss) or the shortest sequence (maximum compression). No single configuration excels at both simultaneously, as visualized in the parameter ablation studies.

Tradeoff between training objective and token count compression across various Unigram parameters.

Algorithmic Robustness

Surprisingly, Unigram is remarkably robust to simplification. Reducing EM steps from five to one or removing complex transformations had negligible impact on final vocabulary quality. This suggests that the core strength of Unigram lies in its basic probabilistic framework rather than its implementation-specific heuristics.

Morphological Superiority

Unlike BPE, which often creates linguistically meaningless "nonsense" tokens to maximize compression, Unigram (and specifically the FSP variant) produces tokens that align more closely with human morphemes. For instance, it more accurately identifies suffixes like "-ed" in English.

Key Results

Efficiency and Compression

Pretoken-based initialization consistently outperformed standard full-text approaches. Furthermore, the simplified FSP method achieved compression rates nearly identical to BPE, effectively closing the historical gap between Unigram and BPE efficiency.

Comparison of Baseline, FSP, and BPE across training tokens, FineWiki tokens, and MorphScore.

Morphological Alignment

Empirical results using the MorphScore benchmark show that Unigram methods substantially outperform BPE. In English, at a 16K vocabulary size, Unigram's boundary recall was significantly higher (0.620) compared to BPE (0.221), indicating a much stronger alignment with linguistic units.

Generalization

While Unigram models training data more effectively, BPE shows a slight advantage in generalizing to completely unseen datasets (out-of-domain) in terms of raw compression, though Unigram remains superior in morphological consistency.

Limitations and Future Work

This study focuses on intrinsic metrics—how well the tokenizer models text—rather than extrinsic downstream LLM performance. Future research should evaluate whether the improved morphological alignment of simplified Unigram variants translates into better performance on complex reasoning or coding tasks. Additionally, the reasons behind BPE's superior out-of-domain generalization warrant further investigation.

Summary updated

BibTeX
@inproceedings{land2026which,
  author = {Sander Land and Yuval Pinter},
  title = {{Which Pieces Does Unigram Tokenization Really Need?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.316},
}
Figure