Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?

Sathvik Nair, Philip ResnikEMNLP 2023 · DOI

Abstract

Large Language Models (LLMs) are increasingly utilized as cognitive proxies for human language processing. However, a fundamental discrepancy exists: LLMs process text via statistically-derived Byte-Pair Encoding (BPE), while humans likely decompose words into meaningful morphemes. This paper investigates whether this tokenization mismatch impacts the predictive power of surprisal for human reading times. Our results indicate that while BPE-based models perform adequately in aggregate, they fail to capture the nuanced cognitive effort associated with complex, segmented words, suggesting that linguistic structure remains a critical component of cognitive realism.

Introduction

A cornerstone of psycholinguistics is the theory that human processing effort is proportional to surprisal—the unexpectedness of a word given its context. Researchers frequently use LLMs to calculate surprisal and correlate it with reading times. However, LLMs "see" words as statistical sub-units (BPE), whereas humans process morphemes (e.g., de- + compos + -ition). This study asks: does this difference matter for modeling human cognition?

Methods

To isolate the effect of tokenization, the authors trained three 5-gram language models on the same training data (a subset of the COCA corpus) using different segmentation strategies:

  1. Orthographic: Whole words only.
  2. BPE: Standard GPT-2 statistical tokenizer.
  3. Morphological: Linguistically valid morphemes via a specialized transducer.

Predictive power was evaluated on the Dundee Corpus (eye-tracking) and the Natural Stories Corpus (self-paced reading) using $\Delta LogLik$—the improvement in regression fit when surprisal is added to baseline features like word length and frequency.

Key Results: The Aggregate Illusion vs. The Split Divergence

In the aggregate, all three tokenization methods improved reading time predictions by a similar margin ($\Delta LogLik \approx 0.01$), with no statistically significant difference between BPE and Morphological models. This explains why previous research has largely overlooked BPE's limitations.

However, a significant divergence appears when focusing on words that the tokenizers actually segment. For these complex words, the BPE-based model's performance drops significantly compared to the morphologically-aware model.

Figure 1: Comparison of predictive power (ΔLogLik) for whole vs. split words across BPE and Morphological models. BPE performance degrades specifically on segmented words.

Cognitive Realism and Surprisal Scaling

Standard cognitive theory suggests that processing effort should scale incrementally with the number of units in a word. Morphological surprisal follows this linear pattern. In contrast, BPE surprisal exhibits a "sharp jump": the surprisal for a word split into two BPE tokens is often nearly identical to one split into four. This lack of linear scaling suggests BPE does not accurately reflect how human effort increases with word complexity.

Figure 2: Distributions of surprisal relative to the number of tokens. Morphological units provide a more consistent, linear relationship with complexity than BPE units.

Furthermore, BPE is "lazy" in English, splitting only about 5% of tokens compared to the 25-44% split by the morphological segmenter. By treating many complex words as single units, BPE misses the internal structure that humans likely process.

Limitations and Future Work

This study was limited to English; the gap between BPE and morphology is expected to be much wider in morphologically rich languages like Turkish. Additionally, while 5-gram models provided experimental control, future research should investigate if the massive scale of modern Transformers mitigates or merely masks this BPE mismatch. The authors urge the NLP community to consider morphologically valid segmentations to support more accurate cognitive research.

Conclusion

While BPE is statistically convenient for training large models, it is not a perfect substitute for linguistic structure in psycholinguistic modeling. For researchers using LLMs to study human language processing, the choice of tokenizer is a critical variable that can obscure the true relationship between surprisal and cognitive effort.

Summary updated

BibTeX
@inproceedings{nair2023words,
  author = {Sathvik Nair and Philip Resnik},
  title = {{Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.752},
}
Figure