Several recent results say that a language model's likelihood and its linguistic ability can point in opposite directions, and that the tokenizer is to blame. Nepali models whose tokenizers reach the lowest pre-training perplexity do worst on downstream tasks. Llama 3 prefers an ungrammatical Japanese sentence to its grammatical twin because the grammatical suffix breaks into raw bytes. These results look like a modern version of an old finding from unsupervised grammar induction, where maximizing corpus likelihood reliably produced bad parse trees. But the modern cases share a confound the old ones did not. Each scores one arbitrary segmentation of the text, one token at a time, and then compares models or sentences that were cut into different numbers of pieces.
We propose to separate the part of this "likelihood–accuracy divergence" that comes from how likelihood is measured from the part that is real. We rescore the known divergence cases with a tokenizer-invariant quantity: the probability of each whole word, summed over all the ways the tokenizer could have spelled it. We then ask whether any divergence survives. For the part that survives, we test a predictor borrowed from grammar induction. It is the spread of word-level surprisal (how unexpected each word is to the model) across a corpus, which predicted grammar quality better than likelihood did for probabilistic grammars. The result is either a correction to a growing narrative about tokenizers, or a sharper and better-supported version of it.
Core Claim
Our claim is that most of the reported tokenizer-driven gap between likelihood and accuracy disappears once likelihood is measured per character or per word and marginalized over segmentations. We expect whatever remains to be predicted better by the variance of word-level marginal surprisal than by any form of mean likelihood. The claim is falsifiable at each rung, and every outcome is informative. If the divergence survives correct measurement, tokenizers really do trade likelihood against competence. If it vanishes, the field has been comparing incomparable numbers.
The motivating evidence comes from three papers. Can Perplexity Predict Finetuning Performance? trains six 24M-parameter Nepali models with different tokenizers and finds that the lowest-perplexity tokenizers (Morpheme+BPE, BPE) are not the best downstream. SentencePiece has the worst perplexity but the best Nep-gLUE average of 0.88, and Morpheme+BPE falls to 0.86 Macro-F1 on pair similarity against 0.96 for SentencePiece. That study compares per-token perplexity across vocabularies that produce different sequence lengths. It also truncates every input at 64 tokens, which removes more text from the finer-grained tokenizers. Inconsistent Tokenizations Cause Language Models to be Perplexed by Japanese Grammar shows that the evidential suffix -sou triggers byte fallback in Llama 3, meaning the text is spelled out as raw UTF-8 bytes. Restricting the test to adjectives that avoid this cuts grammatical-sentence perplexity 28-fold. The paper diagnoses the problem but proposes no correction.
The method comes from Should you marginalize over possible tokenizations?. It estimates the true string probability, summed over segmentations, by importance sampling within whitespace-delimited blocks. It finds the default path loses under 0.5% of log-likelihood on standard English but 1.31% on Basque and up to 2% on transcribed speech. That paper never asks whether the gap changes grammaticality judgments or model rankings. We ask exactly that.
The predictor comes from Variance of Average Surprisal. There, likelihood correlates 0.588 with parsing accuracy against 0.627 for surprisal variance, and variance-based selection gains 4.03 recall points across 25 languages. That work was confined to grammars over gold words. We carry it to neural language models over subwords.
Research Program
The central object is the marginal surprisal of a word $w$ in context $c$:
$$s(w \mid c) = -\log \sum_{T \in \text{tok}(w)} P(T \mid c),$$
Here $\text{tok}(w)$ is the set of token sequences that spell $w$, and $P(T \mid c)$ is the model's probability of emitting that sequence after $c$. We estimate the sum with the block importance sampler of the marginalization paper, using one block per whitespace word. For Japanese, which has no spaces, we use one block per unit from a morphological analyzer.
Summing $s$ over a text and dividing by character count gives a tokenizer-invariant bits-per-character. Our variance statistic takes a different route. We average $s$ over the words of each sentence, then take the variance of those per-sentence averages across the corpus:
$$V = \mathrm{Var}_{\text{sentences}}\Big[\tfrac{1}{|x|}\textstyle\sum_{w \in x} s(w \mid c_w)\Big].$$
Here $|x|$ is the number of words in sentence $x$ and $c_w$ is the preceding context of word $w$. This is the VAS statistic computed in words rather than tokens. It is therefore comparable across tokenizers.
We score every model and every minimal pair along a fixed ladder of measurements. The first rung is per-token perplexity on the default segmentation, which is what the prior papers report. The second is total default-path log-probability normalized by characters. The third is marginalized log-probability normalized by characters. The fourth is the word-level variance statistic $V$.
The drop in divergence from rung one to rung two measures pure length-normalization artifact. The drop from two to three measures segmentation-ambiguity artifact. Whatever divergence remains at rung three is the real phenomenon, and rung four tests whether a distributional statistic captures it.
Baselines
The naive reference point is per-token perplexity on the default tokenization. The Nepali and Japanese papers use exactly this, and it produces the reported inversions. We compute it on identical checkpoints and data, so every other rung differs only in the scoring rule.
The strongest simple competitor is default-path bits-per-character, the standard fix for cross-tokenizer comparison. It costs one forward pass and needs no sampling. This is the baseline most likely to make our marginalization machinery unnecessary. Marginalization "beats" it only if it changes a model ranking or a minimal-pair decision that bits-per-character gets wrong. Matching its correlation with downstream accuracy is not enough.
For the prediction task, the strongest existing linguistic predictor is morpheme-boundary alignment measured with MorphScore. We use the benchmark from Why do language models perform worse for morphologically complex languages?, available for many languages but not, per the file, for Nepali. Where MorphScore does not cover a language, we substitute boundary agreement with a Morfessor segmentation. Our statistic $V$ must exceed alignment's rank correlation with downstream scores across the full model grid to count as better.
The reported number we are trying to explain is SentencePiece's Nep-gLUE average of 0.88 against WordPiece's 0.78 and Morpheme+BPE's pair-similarity 0.86 (Nepali paper). For the Japanese task, the file reports the 28-fold perplexity factor but no minimal-pair accuracy. We found no reported result on Lacuna for Japanese psych-predicate minimal-pair accuracy, so we establish that baseline ourselves.
The key ablation isolates segmentation mass. We compute $V$ from default-path word surprisal instead of marginal surprisal. If it predicts equally well, marginalization contributes nothing to the predictor.
Why not just…?
Why not just report bits-per-character and stop? This is the real threat, and the ladder is built to test it. If rung two already removes the Nepali inversion and the Japanese preference flip, that is itself the finding: the published divergences are normalization artifacts. Marginalization would then be unnecessary except in high-gap regimes. We expect it to matter mostly on minimal pairs, because the difference between the two sentences is concentrated in one fragmented word. That is exactly where a 1–2% corpus-level gap can be decisive. But we treat this expectation as a prediction, not an assumption.
Why not just use byte-level or token-free models? Switching models does not tell us whether the existing tokenizer comparisons were valid, which is the question here. It also carries costs: in the Nepali study, byte-level BPE was slower to train and scored below SentencePiece downstream. The Japanese paper identifies byte fallback as the mechanism of the error, not a cure. We include a byte-level model in the grid as one point, not as the answer.
Why not just scale up? The Japanese inversion appears in 7–10B models, including Llama 3, so scale alone has not removed it. In our framing, scale may shrink the rung-three residual. It cannot fix the rung-one artifact, which is a property of the scoring rule rather than the model.
Why not just pick morphologically aligned tokenizers? The Goldfish study found that alignment does not explain cross-language gaps once byte premiums are controlled. Yet BPE-knockout raised Dutch NER from 83.98 to 87.51 by pruning merges that cross morpheme boundaries. We include knockout-style and Morfessor tokenizers in the grid and use alignment as a competing predictor. That lets the data decide.
Proposed Experiments
The first experiment reproduces the Nepali setup with its confounds removed. We use the same 1.2GB of OSCAR Nepali and the same 24M-parameter architecture, but set the context window in characters rather than tokens, so no tokenizer loses more text to truncation. The grid crosses the six original tokenizers with three vocabulary sizes. We add Unigram variants from Which Pieces Does Unigram Tokenization Really Need? that sit at opposite ends of its likelihood-versus-compression trade-off. Each configuration is trained with three seeds.
Every checkpoint is scored on all four rungs, then fine-tuned on the four Nep-gLUE tasks under an identical tuning budget. The decisive statistic is the Spearman correlation between each rung and downstream Macro-F1 across the grid, with seed-level bootstrap intervals. The original study had only six points and no seeds. A grid of this size can distinguish a sign flip from noise.
The second experiment addresses minimal pairs. We rebuild the Japanese psych-predicate suite and score Llama 2, Llama 3, Mistral, Swallow and Weblab-10B under each rung, reporting pairwise preference accuracy. We separate items that trigger byte fallback from those that do not. The decisive check is whether preference accuracy on byte-fallback items rises to match the well-behaved items at rung two or rung three. We then extend the check to Basque, where the marginalization paper already measured a 1.31% gap. The pairs there would target case suffixes that fragment under the default tokenizer.
The third experiment is a transfer check for the variance statistic. We compute $V$ on the FLORES-200 text of each Nepali and Basque checkpoint before fine-tuning. We ask whether it beats rung-three likelihood and MorphScore as a predictor of downstream rank, mirroring the 0.627-versus-0.588 comparison from grammar induction.
Taken together, the outcomes map directly onto conclusions. If rung two erases the divergence, the tokenizer likelihood paradox is a reporting convention, and we would recommend that papers stop reporting per-token perplexity across tokenizers. If only rung three erases it, the paradox comes from segmentation ambiguity, and marginal scoring should become standard for grammaticality tests. If it survives rung three and $V$ predicts it, the grammar-induction lesson transfers: likelihood genuinely misranks tokenizers, and a cheap distributional statistic can rank them before any fine-tuning.
References
- Can Perplexity Predict Finetuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepali
- Inconsistent Tokenizations Cause Language Models to be Perplexed by Japanese Grammar
- Should you marginalize over possible tokenizations?
- Variance of Average Surprisal: A Better Predictor for Quality of Grammar from Unsupervised PCFG Induction
- Why do language models perform worse for morphologically complex languages?
- BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision
- Which Pieces Does Unigram Tokenization Really Need?