Analyzing Cognitive Plausibility of Subword Tokenization

Lisa Beinborn, Yuval PinterEMNLP 2023 · DOI

Abstract

Subword tokenization is a standard preprocessing step in modern Natural Language Processing (NLP), yet its alignment with human cognitive processes remains under-explored. This paper investigates the "cognitive plausibility" of popular tokenization algorithms—Byte-Pair Encoding (BPE), WordPiece (WPC), and UnigramLM (UNI)—by correlating their "chunkability" metrics with human behavioral data from Lexical Decision Tasks (LDT). Our findings reveal that UnigramLM is less cognitively aligned than BPE or WordPiece, and that multilingual models exhibit a significant "cognitive mismatch" compared to monolingual counterparts. These results suggest that standard engineering-driven tokenization metrics may overlook critical aspects of linguistic processing.

Introduction

In modern NLP, raw text is typically processed using subword tokenization, which breaks words into smaller units (e.g., "seafood" $\rightarrow$ "sea" + "food"). Traditionally, these tokenizers are evaluated based on engineering metrics such as compression efficiency or downstream task performance. However, these mathematical splits often fail to reflect how the human brain processes language. This research addresses the gap between data-driven tokenization and human cognitive reality, asking whether nonsensical splits (e.g., "outfoxed" $\rightarrow$ "out-fo-x-ed") hinder a model's ability to learn semantic meaning.

Methodology

The authors introduce Chunkability, a metric measuring how "whole" a word remains after tokenization, calculated as $1 - \frac{\text{number of tokens}}{\text{number of characters}}$. High scores indicate the word is treated as a near-single unit, while low scores indicate high fragmentation. This metric is correlated with human data from Lexical Decision Tasks (LDT), specifically Response Time (RT) and Accuracy, across English, Dutch, French, and Spanish.

The study compares three dominant algorithms: BPE, WordPiece, and UnigramLM. Data sources include the British, Dutch, and French Lexicon Projects, Spalex for Spanish, and training data from the Leipzig Corpus. Additionally, morphological data from the SIGMORPHON 2022 shared task was used to evaluate alignment with linguistic building blocks.

Results and Analysis

Algorithm Performance and the UnigramLM Paradox

Contrary to previous research suggesting UnigramLM better captures morphology, this study finds it is less cognitively plausible than BPE or WordPiece. As shown in the figure below, BPE and WordPiece consistently show higher correlations with human accuracy and reading times across multiple languages.

Correlations between tokenizer chunkability and human behavioral metrics (Accuracy and Reading Time) for words and non-words across four languages.

Interestingly, for "non-words," the correlations flip. Humans take longer to reject a non-word if a tokenizer assigns it high chunkability (treating it as a "clean" chunk), suggesting that tokenizers and humans share a similar "sense" of what constitutes a valid word structure.

The Curse of Multilinguality

A significant finding is the "cognitive mismatch" in multilingual models. Multilingual models like mBERT use a shared vocabulary that inhibits the tokenizer's ability to reflect human cognitive performance. Monolingual tokenizers are significantly more "human-like" in their splitting behavior compared to their cross-lingual counterparts.

Comparison of Pearson's Correlation for Accuracy and Response Time between monolingual and multilingual BERT models across different languages.

Vocabulary and Morpheme Coverage

For morphologically rich languages, standard vocabulary sizes (e.g., 50,000) are often insufficient. Morpheme coverage continues to increase with vocabulary size, suggesting current models may be under-equipped for complex languages. WordPiece tokens generally provided better coverage of actual derivational morphemes than UnigramLM, reinforcing its cognitive grounding.

Discussion and Conclusion

While this study provides strong evidence for cognitive differences between tokenization strategies, it is limited to four Western European languages. Future work should explore agglutinative languages like Turkish or Finnish, where patterns might differ. Additionally, while the authors attempted to control for word frequency, it remains a confounding factor in both human and machine processing.

The researchers suggest that future NLP development should move toward integrating human cognitive metrics, such as response times, directly into the training or evaluation of tokenizers to create models that process language more naturally.

Summary updated

BibTeX
@inproceedings{beinborn2023analyzing,
  author = {Lisa Beinborn and Yuval Pinter},
  title = {{Analyzing Cognitive Plausibility of Subword Tokenization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.272},
}
Figure