Every modern language model reads text through a tokenizer, a fixed dictionary of word pieces chosen before training by counting frequent character strings. Those pieces often cut across the real units of meaning. "unhappy" can become "unh" + "appy", and in languages that build words by stacking many affixes, roots and grammatical endings get split into opaque fragments. Linguists who study how children learn language describe a related puzzle, the bootstrapping problem: to find the units in a stream of language you need to know the vocabulary, and to know the vocabulary you need the units. Studies of that puzzle find that solving both sides together beats solving them one after the other. Current tokenizers make the opposite choice. They fix the units once, from raw counts, and never let what the model later learns about context change them.
We propose closing that loop. Train a small language model on the current segmentation. Use the model's contextual predictions to re-score how each word should be split and to prune and rebuild the dictionary. Retrain, and repeat a few rounds. The loop is seeded with a cheap unsupervised list of grammatical affixes so that it does not drift into degenerate solutions. The claim is narrow and falsifiable. After several rounds, the dictionary should line up with true morpheme boundaries better than the best one-shot unsupervised pipeline built from the same ingredients, at equal compute, and the gain should show up in downstream morphosyntax for languages that have no hand-built morphological analyzer. If round three is no better than round one, the bootstrapping idea is wrong for tokenization, and we will say so.
Why this gap, and what prior work rules in or out
Three pieces of evidence from the bootstrapping literature motivate the joint approach. Portelance et al. train a grammar induction model jointly with visual grounding and report constituent F1 of 0.90 against 0.42 for a syntax-first pipeline (Portelance et al.). Elsner et al. jointly infer a lexicon and pronunciation variation with a noisy-channel model and raise lexicon F-score from 66.7% to 81.1% when word boundaries are known (Elsner et al.). Snyder et al. jointly segment related Semitic languages and cut error by up to 24% (Snyder et al.).
The same papers also limit what we can promise. Elsner's gain shrinks to 42.6% → 45.7% once word boundaries are unknown, which shows that joint models need a good seed or they stall. All three systems are small Bayesian or grammar models, and none involves a neural language model's tokenizer.
On the tokenization side, the strongest morphology-aware systems depend on supervision we want to remove. MoVoC forbids byte-pair merges from crossing morpheme boundaries supplied by an analyzer or by human annotators, and raises Tigre MorphScore from 74.6% to 83.9% (MoVoC). KinyaBERT builds a two-tier model on a rule-based Kinyarwanda analyzer (KinyaBERT). Two neural methods do let the model shape its own input units, but neither produces discrete morpheme-like units. Charformer learns soft mixtures of contiguous character blocks end to end (Charformer). FACTORIZER shares codebook entries across similar strings while keeping an upstream fixed segmentation (Samuel and Øvrelid).
The closest work is Functional Lexicon in Subword Tokenization. It shows that running byte-pair encoding for only about 50 merges, then filtering by how many contexts each piece appears in, isolates a language's grammatical affixes without supervision. It applies this once, freezes the result, and passes it downstream. We reuse it as our seed. The difference is that we let a language model revise that seed and the rest of the dictionary over several rounds. The contrast between round one and later rounds is the experiment.
Core Claim
A tokenizer can be re-estimated by hard expectation-maximization, an alternating procedure that fixes the segmentation to train the model and then fixes the model to choose the segmentation. In our version the segmentation-scoring step uses a neural language model's contextual probabilities instead of context-free piece frequencies. This contextual signal carries the information that frequency counts lack. Grammatical affixes are predictable from syntactic context, for example agreement, while stems are predictable from topic. Splits that respect this pattern give the text higher probability under a model that has learned context. The circularity is the same one the bootstrapping literature describes. The model needs good units to learn context, and context is needed to find good units. Alternating, with a sensible seed, should climb out of it.
Concretely, for a word string $x$ with sentence context $c$, the segmentation step picks
$$\hat{z} = \arg\max_{z \in S(x)} \sum_i \log p_\theta(z_i \mid z_{<i}, c),$$
where $S(x)$ is the set of ways to split $x$ into pieces from the current dictionary $V$, and $p_\theta$ is the current language model. The dictionary-update step follows the unigram-tokenizer recipe. It removes the pieces whose deletion least reduces the contextual likelihood of the corpus, and adds candidate pieces formed by merges that do not cross the boundaries chosen most often. This is the MoVoC constraint, applied with boundaries the model induced rather than ones an analyzer supplied.
Every candidate segmentation spells the same string, so the scores are comparable. To keep the dictionary from collapsing into whole words, we fix the dictionary size and prune toward that size, as unigram tokenizers already do.
Token embeddings are composed from a small character encoder plus a residual lookup table. A changed dictionary therefore warm-starts instead of forcing a full retrain.
Research Program
We build the loop on small encoder and decoder language models of roughly 50–100M parameters, one per language, plus one multilingual run. The seed dictionary is the 50-merge functional lexicon from Hopton et al. plus a standard byte-pair dictionary at the target size. Each round trains on a fixed token budget, re-segments the corpus with the formula above, and rebuilds the dictionary. We run five rounds and checkpoint after each.
The analysis has three parts. The first is the per-round curve of morpheme boundary F1 against gold. The second is a split of the gains by affix type: frequent grammatical affixes versus rare derivational ones. The third is a stability check. We restart from a corrupted seed to measure how often the loop falls into a degenerate solution, the failure mode Elsner's unknown-boundary result warns about.
We deliberately do not claim to handle root-and-pattern morphology, where a consonant root is interleaved with vowel templates. Our units are still contiguous substrings. Hebrew and Arabic serve only as a stress test where we expect a plateau.
Baselines
The naive reference is a standard byte-pair or unigram tokenizer at the same dictionary size, trained on the same raw text, with the same model trained for the total token budget of all rounds combined. Matching total compute matters. Otherwise any gain from the loop could just be extra training.
The strongest unsupervised competitor is a one-shot pipeline that uses every ingredient we use except feedback. It extracts Hopton et al.'s functional lexicon, turns its affixes into boundary constraints, trains a boundary-constrained byte-pair tokenizer MoVoC-style, and then trains the model once. This is exactly our round one, so it shares data, seed, dictionary size, and architecture. We give it the full compute budget and the same hyperparameter search over merge count (50–350) and productivity threshold that Hopton et al. explored. Hopton et al. report the 50-merge setting as best on SIGMORPHON 2023 but print no per-language F1; no reported result found on Lacuna for a numerical functional-unit F1 on that benchmark, so we will rerun it ourselves. Beating this baseline means a statistically reliable gain in boundary F1 across most of the seven glossing languages, and a downstream gain on at least one morphosyntactic task, at matched compute.
We also run Morfessor-style offline segmentation followed by byte-pair encoding. For Semitic segmentation, Snyder et al. report monolingual F-scores of 63.22 (Arabic) and 59.78 (Hebrew) on their parallel Bible data (Snyder et al.).
Supervised systems serve as ceilings rather than competitors. MoVoC's 83.9% Tigre MorphScore is the bar an unsupervised method would ideally approach. KinyaBERT is the analyzer-based reference on Kinyarwanda.
For the byte-level alternative, we compare against ByT5 and MrT5 at matched parameters and training tokens. MrT5 reports 65.31% XNLI accuracy against 64.72% for ByT5 (MrT5). We use XNLI only as a sanity check for semantic tasks.
The key ablation replaces the contextual model in the segmentation step with a context-free unigram model, so that only frequency drives re-estimation. This isolates whether context is the active ingredient. A further ablation removes the functional-lexicon seed to test whether the seed is what prevents collapse.
Why not just…?
Why not just use the one-shot unsupervised pipeline, a functional lexicon plus constrained byte-pair encoding? This is the real threat. If it works as well as the loop, the loop adds complexity for nothing. The design settles the question directly, because the pipeline is round one. A flat curve from round one to round five, at matched compute, falsifies our claim. Portelance et al.'s 0.90 vs. 0.42 gap suggests a one-shot pipeline can lock in errors, but that was grammar induction with vision, not tokenization, so we treat it as motivation, not proof.
Why not just read raw bytes, as ByT5, CANINE, or MrT5 do? Byte models avoid the dictionary but pay in sequence length. MrT5 needs a learned deletion gate after several layers of full attention to recover speed. In supervised segmentation, Peters and Martins found that Transformers over subwords beat character-level ones (Peters and Martins). Still, CANINE beats mBERT by 2.8 F1 on TyDi QA passage selection (CANINE), so bytes can win on some tasks. Our matched-compute comparison on UD tagging, lemmatization, and MasakhaNER reports both accuracy and tokens per word, so a reader can see the trade.
Why not just enlarge the dictionary or use subword regularization? A larger dictionary inflates the embedding table, which already holds about 23M parameters in BERT-base according to FACTORIZER. Limisiewicz et al. show that changing how dictionary slots are allocated trades word-level syntax against cross-lingual semantics rather than fixing both (Limisiewicz et al.). Subword regularization samples alternative splits from the same frequency-based model, so it adds robustness but no morphological signal. We include a subword-regularized unigram baseline at the same dictionary size to check this directly.
Why not just learn soft tokenization end to end, as Charformer does? Charformer produces continuous mixtures of contiguous blocks, not a discrete dictionary. Its units cannot be inspected or scored against gold boundaries, and they cannot be transferred to a standard model. We want discrete units for exactly those reasons.
Proposed Experiments
Morpheme alignment is measured as unsupervised boundary F1 against gold segmentations. We use the SIGMORPHON 2022 word-level segmentation data, where supervised DeepSPIN-3 reached an F-measure of 97.29, as a reference for what is learnable. We also use the seven SIGMORPHON 2023 glossing languages (Arapaho, Gitksan, Uspanteko, Lezgi, Tsez, Nyangbo, Natügu), where raw text is scarce and analyzers are absent. In addition, we report MorphScore on Tigre and Amharic with MoVoC's data and Hebrew/Arabic boundary F-score on Snyder et al.'s corpus as the non-concatenative stress test.
Downstream evaluation covers UD POS tagging, dependency parsing, and lemmatization (BLEX) on Turkish, Czech, Arabic, and Scottish Gaelic. It also covers MasakhaNER for Amharic and Kinyarwanda, and Kinyarwanda news classification against KinyaBERT. We report fertility (tokens per word) alongside every result.
The decisive checks are three. The per-round boundary F1 curve must rise past round one beyond seed-to-seed variance. The contextual-scoring ablation must lose most of that rise. Downstream morphosyntax must improve over the one-shot pipeline without higher fertility. Passing the first two but failing the third would mean the loop finds better morphemes that models do not need. That is a useful negative result for the debate over whether subword reuse or byte fragmentation matters more.
References
- Reframing linguistic bootstrapping as joint inference using visually-grounded grammar induction models
- Bootstrapping a Unified Model of Lexical and Phonetic Acquisition
- Unsupervised Multilingual Learning for Morphological Segmentation
- MoVoC: Morphology-Aware Subword Construction for Ge’ez Script Languages
- KinyaBERT: a Morphology-aware Kinyarwanda Language Model
- Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
- Tokenization with Factorized Subword Encoding
- Functional Lexicon in Subword Tokenization
- MrT5: Dynamic Token Merging for Efficient Byte-level Language Models
- Beyond Characters: Subword-level Morpheme Segmentation
- Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
- Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages