Introduction
Modern language models (LMs) are frequently used in computational linguistics to model human language processing. A key metric in this field is surprisal—the negative log probability of a word—which serves as a proxy for the cognitive effort required for a human to read that word. However, a subtle but fundamental mathematical flaw exists in how these probabilities are typically calculated. Most popular LMs, such as GPT-2 and Llama, utilize tokenizers that attach whitespaces to the beginning of subword tokens (e.g., _apple). This paper, presented at EMNLP 2024, demonstrates that this "leading whitespace" convention creates a mathematical inconsistency that misallocates probability mass and distorts surprisal values.
The Leading Whitespace Confound
The core issue lies in the fact that LMs predict subword tokens rather than full words. Under the standard leading-whitespace scheme, a model does not "finish" a word until it predicts the whitespace at the start of the subsequent word. Consequently, at any given point, the model may assign high probability to both a complete word (e.g., "mat") and a longer word that begins with the same prefix (e.g., "matron").
Mathematically, this violates the Kolmogorov axiom ($P(\Omega) = 1$). Because there is no explicit "stop" token for the current word, the total probability mass for all possible words at a given position can exceed 100%. This "probability leak" means that the difficulty of ending a word is often incorrectly carried over to the processing of the following word in the sequence, as seen in the high surprisal for the token _in when it follows _mat compared to when it follows ron in the figure above.
Proposed Solution: Whitespace-Trailing Decoding
To rectify this, the authors propose Whitespace-Trailing (WT) decoding. This method mathematically shifts the probability of the leading whitespace from the start of the next word to the end of the current word. By re-weighting word probabilities based on the likelihood that the next token will be a whitespace-beginning token, the model's internal state is aligned with the reality of human reading experiments (like self-paced reading), where the reader knows a word has ended once they move to the next one.
Experimental Setup
The researchers evaluated the impact of WT decoding using several psycholinguistic datasets, including the Provo Corpus, Natural Stories, and the Dundee Corpus. They also focused on Garden-Path Sentences, which are designed to induce temporary syntactic ambiguity (e.g., "After the doctor left the room turned very dark..."). They tested these effects across various models, including GPT-2 variants and the Pythia suite, using Linear Mixed Effects Regression (LMER) to compare the predictive power of standard leading-whitespace surprisal versus the corrected WT surprisal.
Key Results
The study revealed that the whitespace confound significantly impacts the interpretation of linguistic phenomena. In garden-path sentences, applying the WT correction resulted in notably lower estimates of processing difficulty than previously reported. This suggests that prior research may have overestimated the "struggle" LMs face with these complex constructions due to the misallocation of surprisal.
Interestingly, the fit to naturalistic reading data was more complex. In some cases, WT decoding actually led to poorer fits. The authors hypothesize that the original "flawed" method might have been unintentionally masking other model weaknesses or capturing a genuine human uncertainty about word boundaries that the "perfect" mathematical fix removes.
Conclusion and Future Directions
The "leading whitespace" confound is a critical consideration for researchers using LMs to study human cognition. While WT decoding provides a mathematical "patch" for existing models, the authors suggest that future language models should be trained with tokenizers that use trailing whitespaces by default. This would ensure mathematical consistency from the start and provide a more accurate foundation for both cognitive modeling and downstream NLP interpretability.
Summary updated