Tokenizer or Capacity? Disentangling the Monolingual Advantage on Nonce Morphology

Language models trained on a single language often beat much larger multilingual models on that language's benchmarks. Two explanations are usually given together, and nobody has separated them. The first is that the monolingual model has its own tokenizer, the component that cuts text into the units the model sees. A Turkish-only tokenizer is more likely to cut words at real suffix boundaries. The second is that the model spends all of its parameters on one grammar rather than spreading them across a hundred. A third possibility makes the question urgent: the advantage may not involve learning grammar at all. The monolingual model may simply hold a denser memorized table of real word forms. Tests built on invented words show that current models mostly retrieve stored exemplars and do not apply rules.

We propose a controlled pretraining study in Turkish and Finnish, two languages where one word can stack many suffixes. The study crosses tokenizer type with pretraining scope (monolingual versus multilingual, at matched compute) and evaluates every model on invented-root word formation and on inflection of unseen base words. The central claim we test is specific and falsifiable. A tokenizer that splits words along morpheme boundaries helps a model generalize to new words only if it is used during pretraining, not when it is applied afterwards at inference time. Once the tokenizer is held fixed, most of the "monolingual advantage" on invented words should disappear. If this holds, the field's main negative result on morpheme-aligned tokens has been misread. If it fails, we have the first clean evidence that capacity concentration, rather than segmentation, drives morphological generalization.

Why this question, and why now

The motivating tension is between two bodies of evidence that never touch. Evaluating Morphological Compositional Generalization in Large Language Models built Turkish and Finnish tests with two tasks. Productivity asks the model to produce the word for a root plus a set of affixes. Systematicity asks it to judge whether a derived form is valid. Both tasks use real roots and nonce roots, meaning plausible invented roots. Model accuracy collapses as the number of suffixes grows. GPT-4 trails humans by about 43% (Turkish) and 51% (Finnish) on productivity. It reaches only about 51% coherence on Turkish nonce-root systematicity, against about 98% for humans; coherence means getting every derivation for a given affix set right. Crucially, that paper fed general-purpose models pre-segmented, morpheme-aligned input and saw no significant gain. It concluded that tokenization is not the main cause.

Meanwhile, The Token Tax finds that fertility, the average number of tokens per word, explains up to 50% of accuracy variance across African languages. SeaLLMs shows large downstream gains from adding native subwords to the vocabulary. Native models such as Jais (ArabicMMLU) and NorGLM (NLEBench+NorGLM) beat larger general models. These papers motivate the proposal: they establish the advantage. They do not tell us whether it reflects rule learning. That gap matters because (Un)solving Morphological Inflection showed that standard benchmarks leak base words. When those leaks are removed, average accuracy across 90 languages falls from 0.88 to 0.58.

Our reading of the null result in Ismayilzada et al. is that it contains a train–test mismatch. Those models were pretrained on frequency-driven segmentations. Handing them morpheme-aligned pieces at test time presents token sequences they rarely saw, so it is unsurprising that this did not help. Revisiting subword tokenization: affixal negation points the same way from the other side. Hyphenating English words raised Llama-2's negation accuracy from about 77% to about 96%, but models still failed on nonce words. Surface fixes help retrieval; they do not supply rules. No study has asked what happens when alignment is present from the first step of pretraining.

Core Claim

The hypothesis has two parts. First, aligning tokens with morphemes during pretraining makes each suffix appear as one consistent unit across thousands of different stems. That raises the type frequency of the suffix, meaning the number of distinct stems it attaches to. Abstraction should track type frequency, and rote retrieval should track raw token frequency. Derivational Morphology Reveals Analogical Generalization found that GPT-J's suffix choices on English nonce adjectives correlate with training-data token statistics at r = 0.995, and that GPT-4 is less human-like because it is even more frequency-driven. We therefore predict that aligned-pretraining models will shift from token-frequency-driven choices toward type-frequency-driven ones.

Second, once tokenizer and target-language exposure are matched, monolingual and multilingual models should be close on nonce-root generalization. If so, the advantage reported in the native-model literature is mostly a segmentation and exposure effect, not a capacity-dilution effect.

Research Program

We pretrain decoder-only models at two sizes, roughly 150M and 1B parameters. These sizes are well under the data ceiling that limited NorGLM's scaling, since Turkish and Finnish web text supports several billion tokens. Every run in a size class uses the same architecture, optimizer, token budget in bytes, and learning-rate schedule. We fix bytes rather than tokens so that tokenizers with higher fertility are not given more text.

The tokenizer factor has four levels:

  • Monolingual BPE: byte-pair encoding, which merges frequent character pairs, trained on the target language alone.
  • Multilingual BPE: byte-pair encoding trained on a ten-language mixture, so that the target language receives an inherited, poorly fitted vocabulary.
  • Morpheme-aligned: a tokenizer that first splits words with a morphological analyzer and then applies BPE within the pieces, in the spirit of the analyzer-based design of KinyaBERT but without its two-level architecture.
  • Byte-level: no learned vocabulary, following the tokenization-free motivation of CANINE.

The scope factor has three levels:

  • Monolingual: the target language only.
  • Compute-matched multilingual: the target language is 10% of a ten-language mixture with the same total bytes. This tests dilution under a fixed budget.
  • Exposure-matched multilingual: the same target-language bytes as the monolingual run, plus nine other languages. This uses more total compute, but the target language is seen exactly as often, which isolates parameter sharing from exposure.

We run the full grid at the small size for both languages. At 1B we run a subset: the monolingual/multilingual contrast under the monolingual-BPE and morpheme-aligned tokenizers. Before evaluation, we check that every nonce root is absent from each pretraining corpus.

On each model we compute three diagnostics:

  • A per-tokenizer "alien" rate: the share of segmentations that cut across morpheme boundaries. This extends the English-only labeller of Evaluating Subword Tokenization to Turkish and Finnish using UniMorph analyses.
  • Affix consistency: how often a given suffix maps to the same token sequence across stems.
  • A type-versus-token frequency regression. We own the corpora, so for each suffix choice we can compute the exact type and token frequencies the model saw. This removes the estimation step Hofmann et al. needed for closed models.

Baselines

The naive reference point is the multilingual-BPE, compute-matched multilingual model. It is our small-scale stand-in for a general-purpose model, and we report it alongside the random baseline of each task. Ismayilzada et al. note that most LLMs barely beat random on nonce roots. Any condition that does not clearly separate from this cell has failed.

The strongest existing system is GPT-4 as evaluated by Ismayilzada et al.: about 51% Turkish nonce-root systematicity coherence, and roughly 43% (Turkish) and 51% (Finnish) below human productivity. We cannot retrain it. We use its published numbers as an external ceiling reference under the same prompts and metrics, and we re-run the released prompts on our models. Our 1B models are not expected to beat GPT-4 outright. Matching its coherence at three orders of magnitude fewer parameters would be meaningful.

For inflection on unseen base words, the reference points are as follows. Goldman et al. report a macro-average of 0.58 across 90 languages on the lemma-split. SIGMORPHON–UniMorph 2022 Shared Task 0 reports no system above 72% average accuracy. We evaluate on the Turkish and Finnish portions of those splits after identical light fine-tuning of every model. Beating the trained-from-scratch character transducers from those tasks on the unseen-lemma and unseen-feature-set conditions counts as evidence that pretraining helped.

The strongest real competitor is vocabulary expansion of a multilingual model, as in SeaLLMs. We implement it by taking the compute-matched multilingual-BPE model, adding monolingual-BPE tokens, and continuing pretraining on target-language text. The continued pretraining is given the same byte budget as the difference between our scope conditions. Its tuning is matched to the other runs.

The ablation that isolates our key ingredient swaps tokenizers at inference only. Models pretrained with monolingual BPE are given morpheme-aligned input at test time, and morpheme-aligned models are given BPE input. Our claim predicts an asymmetry. Alignment applied at test time should not help, which replicates Ismayilzada et al. Alignment during pretraining should help.

Why not just…?

Why not just evaluate existing native models like a Turkish BERT, Jais, or NorGLM against GPT-4? This is cheap, and we will do it as a pilot. However, any difference it shows is confounded. Those models differ from multilingual ones in data, size, tokenizer, and instruction tuning all at once. The ArabicMMLU finding that Arabic models do better with English instructions shows how much tuning artifacts can move scores. A pilot can show that the advantage exists on nonce morphology. Only the controlled grid can say why.

Why not just segment the input at inference, with hyphens or spaces between morphemes? For English single-affix negation this works on real words: about 77% to about 96% for Llama-2 (Truong et al.). It does not work on nonce words. For Turkish and Finnish affix stacking, Ismayilzada et al. found no significant gain. Our inference-swap ablation tests this directly on models whose training tokenizer we control.

Why not just expand the vocabulary of a multilingual model? This is a serious threat. It may recover most of the benefit of aligned pretraining at a fraction of the cost. We run it as a full baseline. If it matches the morpheme-aligned monolingual model on nonce coherence and on how sharply accuracy collapses with more suffixes, our recommendation becomes "adapt, don't pretrain." We would report that outcome as the result. Against it stands the finding of Batsuren et al. that frequency-driven vocabularies keep many alien segmentations even at 40–50K tokens. We will measure whether the expanded vocabulary's alien rate predicts its nonce performance.

Why not just scale? The published evidence says scale does not fix this. GPT-4 collapses on long suffix chains, and Hofmann et al. find it less human-like than GPT-J. We include two sizes to estimate whether the effect of the tokenizer factor shrinks or grows with scale.

Proposed Experiments and Decisive Checks

The primary outcomes are productivity accuracy and systematicity coherence on Ismayilzada et al.'s Turkish (1–7 suffixes) and Finnish (1–6 suffixes) sets, separately for real and nonce roots. We also summarize each model by a collapse slope $\beta$, fitted as

$$ \text{acc}(k) = \alpha + \beta k, $$

where $\text{acc}(k)$ is nonce-root productivity accuracy on words with $k$ bound morphemes. A shallower (less negative) $\beta$ means less collapse. Secondary outcomes are lemma-split and unseen-feature-set inflection accuracy, the alien rate, and the type-versus-token regression coefficients.

Three results would decide the question. First, if pretraining-time alignment reduces nonce collapse while test-time alignment does not, the null result of Ismayilzada et al. is reinterpreted as a train–test mismatch. Second, if the monolingual-versus-exposure-matched-multilingual gap on nonce coherence shrinks toward zero under a fixed tokenizer, the monolingual advantage is a segmentation and exposure effect. A gap that persists would be the first direct evidence for capacity dilution in morphology. Third, if aligned models' suffix choices load on type frequency rather than token frequency, that links the tokenizer to the exemplar-versus-rule account of Hofmann et al. If none of these hold, and every cell collapses at the same rate, then the collapse is a property of transformer pretraining that is independent of both segmentation and scope. That would be a clean negative result and would redirect effort toward architecture.