Measuring the Tokenizer as a Method Factor in Multilingual Evaluation

When a multilingual model scores 30 points lower on Yoruba than on English, we usually read that as a fact about the model's knowledge of Yoruba. Survey researchers would see a familiar mistake. In their field, a score produced by a single instrument mixes the thing being measured (the "trait") with the quirks of the instrument (the "method"). The share of score variance that comes from the instrument is called common method variance. The tokenizer, the component that cuts text into the units a model reads, is exactly such an instrument. It is shared by every item and task in a language, and it behaves very differently across scripts. The evidence is strong. The Token Tax reports that tokens-per-word ("fertility") explains up to 50% of the variance in accuracy on AfriMMLU. Why do language models perform worse for morphologically complex languages? argues the gap disappears once training data is scaled by the "byte premium", which is the extra UTF-8 bytes non-Latin scripts need per character. Both claims are correlational, and they disagree about the cause.

We propose to borrow the strongest tool survey research has for this problem. That tool is not the widely used Harman single-factor check but the multitrait–multimethod design: measure the same traits with several deliberately different instruments, then fit a latent-factor model that separates trait variance from method variance. Here the "methods" are input segmentations applied to otherwise identical small language models. We use a subword tokenizer, raw bytes, bytes followed by a bank of parallel convolutions of different widths (a multi-scale convolutional front end), and a CANINE-style single strided convolution. Crossing these with two training-data regimes, one equal in bytes and one scaled by the byte premium, turns the fertility-versus-data-volume debate into a controlled experiment. It also tests whether multi-scale convolution in particular reduces tokenizer-induced method variance.

Core Claim

Cross-language performance gaps on standard benchmarks contain a separable, estimable method component attributable to input segmentation. Its size can be identified only by varying the segmentation while holding model, data, and compute fixed. Observational regressions on fertility cannot do this, and neither can post-hoc single-factor tests. We further predict that a byte-level encoder with a multi-scale convolution bank absorbs a larger share of this method component than either subword or plain-byte encoders, and that the benefit concentrates in high-byte-premium scripts. If this prediction holds, part of the byte premium can be addressed through architecture rather than data volume.

For a language $l$, task $t$, and segmentation method $m$, we model the standardized score as

$$y_{ltm} = \lambda_{t}\,T_{lt} + \mu_{m}\,M_{lm} + \varepsilon_{ltm},$$

where $T_{lt}$ is the latent capability of the model family on task $t$ in language $l$, $M_{lm}$ is a latent method factor for how segmentation $m$ treats language $l$, $\lambda_t$ and $\mu_m$ are loadings, and $\varepsilon_{ltm}$ is residual noise with its own variance. This is the confirmatory structure used in survey work, and it mirrors the explicit noise term that Multilingual Factor Analysis showed matters for cross-lingual alignment. That paper modeled shared concepts plus language-specific noise over static word vectors. We apply the same separation to evaluation scores, with the segmentation as the source of structured noise. The quantity we report is the method share, $\mu_m^2\operatorname{Var}(M)/\operatorname{Var}(y)$, for each segmentation and data regime.

How This Relates to Prior Work

Three groups of papers play different roles.

The first group motivates the proposal. The Token Tax regresses accuracy on fertility across ten existing LLMs. It cannot separate fertility from the training-data volume and script that travel with it, because every model uses its own tokenizer and its own data. Why do language models perform worse for morphologically complex languages? provides the counterweight. Its MorphScore analysis finds that agglutinative languages align better with morphemes (66.3%) than fusional ones (53.3%), and that byte-premium-scaled data removes the typology gap. That paper constrains us: morpheme alignment should not be assumed to be the method channel, and data volume must be a manipulated factor rather than a nuisance.

The second group supplies the architectures we treat as instruments. Integrating Multi-scale Contextualized Information for Byte-based NMT splits byte hidden states across parallel 1D convolutions with kernels {1, 3, 5, …} before attention. It reaches 24.30 SacreBLEU on TED-59, against 23.63 for a plain byte Transformer, and 20.01 versus 2.79 BLEU against a subword model on zero-shot Medical-domain WMT19 De→En. CANINE downsamples characters with one strided convolution and beats mBERT by 2.8 F1 on TyDi QA passage selection. IS-BERT shows that multi-scale convolutions over encoder outputs help sentence embeddings. None of these papers asks how much of the cross-language variance in its results is attributable to the input method rather than to capability. Each reports a single average, which is exactly what our decomposition takes apart.

The third group supplies the measurement method and its known weakness. Harman's single-factor test, used as a gatekeeper in, for example, High-performance HR practices and employee commitment and Influence of Quality Development Over College Students' Entrepreneurial Competency, only flags method variance when one factor exceeds 50% of total variance. Those studies also note that single-source, single-occasion designs cannot separate method from trait. That limitation is our entry point. In NLP, unlike in surveys, we can cheaply re-measure the same traits with a different instrument.

Research Program

We would pretrain a family of small encoder–decoder models that are identical in depth, width, attention, and optimizer. They would differ only in the input front end, with four levels. The first is a SentencePiece subword vocabulary. The second is raw UTF-8 bytes. The third is bytes plus the multi-scale convolution module of Huang and Feng, placed before attention. The fourth is bytes plus a CANINE-style single strided convolution. Parameter counts would be matched by adjusting embedding size, and the byte variants would be matched in FLOPs by sequence-length handling.

Each front end is trained under two data regimes on a common multilingual corpus. In the first, every language gets the same number of bytes. In the second, each language's budget is scaled by its byte premium measured on FLORES-200, following the Goldfish protocol of Why do language models perform worse…. The 4×2 grid, with three seeds per cell, yields 24 models.

Each model is fine-tuned and evaluated on three trait indicators per language where coverage overlaps. These are TyDi QA passage selection and minimal span (11 languages), MasakhaNER (10 African languages), and TED-59 many-to-English translation restricted to the overlapping languages. We then fit the factor model above by maximum likelihood, with seeds as replicate measurements, and compare it against nested models without the method factor.

As an observational complement, we fit the same model to the published per-language AfriMMLU results from The Token Tax. The ten LLMs serve as uncontrolled methods, and fertility, byte premium, and MorphScore serve as marker indicators of $M$. This tells us whether a method factor is visible in the wild and how far the observational estimate departs from the controlled one.

Baselines

The do-nothing reference is the raw per-language gap to English on each benchmark, read as capability. It implicitly sets the method share to zero. Any nonzero method share we identify with confidence intervals excluding zero counts as evidence against this reading.

The survey-research default is Harman's single-factor test run on the same language-by-task score matrix, under the 50% first-factor rule used in the cited organizational studies. We run it on identical data. The proposal beats it if a planted method component that Harman's test passes as acceptable is recovered by the multitrait–multimethod model. This is checked by simulation below and on real data.

The strongest existing analysis is the fertility regression of The Token Tax, which reports that fertility explains up to 50% of AfriMMLU accuracy variance and that each extra token per word costs up to 18 points. We reproduce it on our grid by regressing scores on per-language fertility within each data regime. The proposal beats it if the controlled method share differs materially from the fertility R² in a way the regression cannot detect. For example, fertility might "explain" variance under equal-byte data that vanishes under byte-premium scaling. That pattern would show fertility was partly a proxy for data volume. We also include the mixed-effects byte-premium model from Why do language models perform worse… as a second analytic competitor.

The strongest architectural baseline is the multi-scale module itself, compared against the plain byte Transformer. On TED-59 we target the reported 24.30 average SacreBLEU for multi-scale contextualization against 23.63 for the plain byte model, as reported by Huang and Feng. Our models are smaller, so the comparison is relative ordering within our grid, not absolute BLEU.

For TyDi QA, the reference is CANINE's reported +2.8 F1 over mBERT on passage selection. For MasakhaNER, no reported result was found on Lacuna for CANINE's numbers.

The key ablation replaces the multi-scale bank with a single kernel-3 convolution of matched parameters, plus an identity-only variant. This isolates whether any reduction in method share comes from multiple receptive fields or merely from adding local convolution. All model comparisons use the same data, identical fine-tuning budgets, and a fixed hyperparameter sweep of equal size per front end, reported across three seeds.

Why not just…?

Why not just regress scores on fertility, as The Token Tax did? Fertility is one observed proxy, and in existing models it is entangled with data volume, script, and model identity. A regression cannot tell a causal tokenizer effect from a correlated data shortage, which is precisely the disagreement between the two cited papers. Our design manipulates segmentation and data volume independently, so the fertility regression becomes a special case we can check against ground truth. If the regression's estimate matches the controlled method share in both data regimes, the simpler analysis was adequate, and we would report that.

Why not just scale the data by the byte premium and skip architecture entirely? This is a real threat, and it is one arm of our grid. If subword method share drops to near zero under byte-premium-scaled data, the data explanation wins for these benchmarks. The Token Tax's cost estimate ($105M to $420M for a 2× fertility increase) and the scarcity of digitized text for languages like Aymara (Tokenization as Cultural Erasure) make the practical question whether a front end can reach the same method share at the equal-byte budget. The grid answers that directly.

Why not just use raw bytes and drop the multi-scale bank? Plain bytes remove the vocabulary but introduce low information per unit. Huang and Feng's 0.67 SacreBLEU gain on TED-59 is modest, so the multi-scale bank may matter little for method share. The plain-byte and single-kernel arms test this. If they match the multi-scale model's method share within seed variance, we conclude the benefit lies in removing subwords, not in multi-granular convolution.

Why not just run Harman's test on benchmark scores? It only fires when one factor dominates more than half the variance, and it gives no estimate of partial method effects. The simulation check below quantifies how often it misses realistic method variance.

Proposed Experiments and Decisive Checks

The first check is a planted-variance simulation. We inject a known method component, at method shares from 5% to 40%, into synthetic score matrices shaped like ours. We then verify that the factor model recovers it and document the range over which Harman's test reports "no bias". If the model cannot recover planted shares within ±5 points at our sample size, the program stops, because the design is underpowered.

The second check is the central contrast: the method share for subword versus byte front ends under equal-byte data, with seed-bootstrap intervals.

The third check tests whether the multi-scale front end reduces method share more than the plain-byte and single-kernel ablations. We test this separately for languages above and below the median byte premium. The prediction is that the gain appears only above the median, and a uniform or absent effect falsifies it.

The fourth check tests the interaction between data regime and front end. If byte-premium scaling and the multi-scale front end each shrink method share and are partly substitutable, architecture offers a cheaper route. If only data scaling helps, the architectural claim fails.

The final check fits the model to AfriMMLU and compares the observational method-share estimate with the controlled one. This tells practitioners how much to trust fertility-style diagnoses of published leaderboards.