Large language models read text as subword tokens, the chunks produced by a frequency-based compression procedure that runs once before training. For English this usually works. For morphologically rich languages such as Amharic, Basque or Aymara, the chunks cut across meaningful units, inflate the number of tokens per word, and leave the model holding fragments it cannot reliably reassemble. Every current fix is "open-loop": the tokenizer decides once, and the model never reports back. This includes uniform character splitting, random re-splitting, and frequency-driven merging. This note proposes the opposite. In a first pass, the model's own hidden states reveal which words it failed to put together. Only those words are then re-split into finer units, under a fixed budget of extra tokens, and the model reads the input again.
The motivation comes from information theory, which treats the feedback question carefully. On a single link, perfect feedback does not increase how much information can get through. It helps by telling the sender where the receiver went wrong, so effort can go there and nowhere else. In networks where users share and interfere over a common medium, feedback does much more. Work on interference channels shows the gain is largest, and grows without bound at high signal quality, in an intermediate regime where interference is "too strong to ignore, too weak to decode" (Feedback Capacity of the Gaussian Interference Channel to within 2 Bits). We borrow two lessons, as a design principle and a falsifiable prediction rather than a theorem. The first is to spend extra resources only where the receiver signals failure. The second is to expect the payoff to peak in the intermediate regime.
Core Claim
The claim has two parts. First, at a fixed budget of extra input tokens, re-splitting the words that a cheap probe on the model's hidden states flags as poorly composed will beat two alternatives on morphologically rich low-resource languages: re-splitting randomly chosen words, and re-splitting every word into characters. Second, the gain will not grow steadily with token fertility (tokens per word). It will be small for English-like languages, where words are already whole tokens. It will be largest for languages of intermediate fertility, whose words are split into a few misleading pieces. It will shrink again for scripts that already fall back to near-character or byte tokens, where there is little finer splitting left to request. If the second prediction fails and gains rise monotonically with fertility, the interference-channel lens adds nothing. The method would then reduce to "more characters for harder languages," and we will report that.
Relation to Prior Work
Two papers motivate the gap. Tokenization and the Noiseless Channel models the tokenizer as an encoder sending symbols to the model. It shows that a token-distribution statistic, Rényi efficiency, correlates with translation quality (Pearson 0.78 on WMT). Its channel, however, is point-to-point and has no feedback. Two Counterexamples to Tokenization and the Noiseless Channel then shows that such open-loop statistics can be gamed. Duplicating tokens raises efficiency from 0.474 to 0.648 while IWSLT14 De→En BLEU falls from 33.74 to 31.32. We take both results as the argument for using a signal from the model itself rather than from token statistics.
Two papers supply the enabling facts. Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations shows that instruction-tuned models still work on non-canonical splits. Qwen-2.5-7B-Instruct keeps 93.4% of its performance under random splits and 90.8% under character-level input. Forcing right-aligned digit groups improved 10-digit arithmetic by +33.7%. Those alternative splits are chosen by fixed rules applied to the whole input; ours are chosen word by word from the model's own failure signal. Tokenization Falling Short: The Curse of Tokenization documents why a failure signal should exist. The vector for "assignment" has cosine similarity of only 0.21 with "assign"+"ment", and both Llama-3-8B and 70B hit the same wall as words get longer. We turn this gap into a per-word detector.
Two papers set constraints. Retrofitting Large Language Models with Dynamic Tokenization adapts token boundaries at inference, but it merges by batch frequency, mainly to shorten sequences. It shortens XNLI inputs by 22.5% at under 2% accuracy loss. Our method goes the other direction, selectively lengthening where the model struggles, and we treat that paper's trade-off as the efficiency frontier we must stay near. The Token Tax reports that fertility explains up to 50% of accuracy variance on AfriMMLU. That gives us the fertility axis along which the regime prediction is tested. Interference Channels with Rate-Limited Feedback shows that feedback gains survive when feedback is limited in rate. This motivates our question of how few flagged words are enough, which we treat as the feedback "rate".
Research Program
The system has three components, all built on frozen open models such as Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct, the models studied in Broken Tokens.
The first component is the feedback signal. We train a small linear probe that reads the hidden state at a word's last token from a middle layer and tries to predict the word's character sequence, scored as character-level reconstruction loss. Words the probe cannot spell back are words the model has not composed. Training uses unlabeled text in the target languages, and the model's weights are never touched. As a training-free variant, we use the model's surprisal on the word's non-initial tokens. We keep the probe as primary for a specific reason: the Japanese byte-fallback results of Gambardella et al. suggest that surprisal can be misleadingly low on pathological splits.
The second component is re-segmentation under a budget. We rank words by the probe score and re-split the top-ranked words, either into characters or into BPE-dropout variants (sub-splits of each token drawn by randomly skipping merges), until the extra tokens reach a budget $B$. $B$ is expressed as a fraction of the original sequence length. The model then reads the revised input and answers.
The third component is the analysis. We sweep $B$ to trace an accuracy-versus-length curve. We bin languages by measured fertility to test the regime prediction. We also check whether flagged words concentrate on known morphological sites, such as case and agreement affixes or negation markers, using the misalignment patterns reported by Tokenization as Cultural Erasure for Aymara as a qualitative guide.
Baselines
The do-nothing reference is canonical tokenization on the same frozen model with the same prompt. The file contains no reported result for Qwen-2.5-7B or Llama-3.1-8B on AfriMMLU, AfriXNLI or AfriMGSM, so we measure these ourselves. Benchmark-level context comes from IrokoBench: GPT-4o averaged 59.0% across African-language tasks, and Gemma 2 27B reached 63% of that (IrokoBench).
The strongest same-model competitor is whole-input character-level or random re-tokenization, following Broken Tokens?. It is the strongest because it already exploits the same latent robustness without any selection. Its reported numbers are 90.8% retention under character input and 93.4% under random splits for Qwen-2.5-7B-Instruct. We run it on identical prompts and report it at its natural length cost. We also report a length-capped version that applies character splitting left to right until budget $B$ is reached. Our method beats it if it reaches higher accuracy at equal added length, or equal accuracy at clearly lower length.
The strongest external competitor on African-language benchmarks is translate-test. IrokoBench reports gains of +17.6% for Gemma 2 27B and +21.0% for LLaMa 3.1 70B. We run it with the same base model and a fixed translation system. We do not expect to beat it in raw accuracy on knowledge questions. The meaningful comparisons are whether our method adds on top of it when applied to the source-language input in a hybrid setup, and whether it beats translate-test on the original-language sub-tasks where translation is lossy.
The ablation that isolates the key ingredient is random-word re-segmentation at exactly the same budget $B$ and the same splitting rule. It differs from our method only in that the words are chosen without feedback. We also run an oracle that re-splits the words whose re-splitting most improves the gold answer's likelihood, which gives an upper bound on what better selection could achieve. All methods share prompts, decoding settings and models. The probe's only tuning is choosing the layer on a held-out development split.
Why not just…?
Why not just use a tokenization-free model? CANINE beats mBERT by 2.8 F1 on TyDi QA passage selection and handles Amharic where mBERT fails. However, it is a pretrained encoder that relies on 4x downsampling, and it does nothing for the deployed decoder models people actually use. Our method costs a probe and a second forward pass. It is complementary to, not a substitute for, retraining without tokens.
Why not just split everything into characters? This is the real threat. If whole-input character splitting matches our accuracy at an acceptable length cost, the selection step is not worth having. The length-capped character baseline and the $B$ sweep settle this directly. Broken Tokens? already reports that heavier fragmentation hurts long-context tasks, so we expect selective splitting to dominate as inputs get longer. We will report the outcome either way.
Why not just ensemble several random tokenizations and vote? BPE-dropout-style sampling is cheap and helped on structural tasks in Tokenization Falling Short. Our method uses two forward passes, so we compare against a two-sample vote and also show the vote's curve as samples increase. If a $k$-sample vote overtakes us at equal compute, the gain comes from averaging, not from feedback, and the claim fails.
Why not just scale the model? Chai et al. show the long-word wall persists from 8B to 70B, and The Token Tax finds fertility still predicts accuracy across ten models. We repeat our main result at two model sizes to check that the gain does not disappear with scale.
Proposed Experiments
The main evaluation runs on AfriMMLU, AfriXNLI and AfriMGSM from IrokoBench across their 17 languages, with English as control. Diagnostics come from TKEval's character-counting, common-substring and typo tasks. A shortening check on XNLI compares against Feher et al.'s efficiency frontier. Metrics are accuracy (or F1), added tokens per input, and wall-clock prefill cost.
Four checks decide the outcome. The first is whether feedback-selected re-splitting beats random selection at matched $B$ on the morphologically rich languages, with confidence intervals from per-question bootstrap. Without this, the work stops. The second is whether the probe score predicts per-question errors under canonical tokenization, measured by AUROC (area under the curve of how well the score ranks wrong answers above right ones). This tests whether the "embedding gap" is detectable word by word. The third is whether the gain, plotted against language fertility, peaks at intermediate fertility as predicted, rather than rising monotonically. The fourth is whether the gain saturates at small $B$, meaning a few flagged words capture most of the benefit. That would be the practical counterpart of the rate-limited feedback result, and it would make the method cheap enough to deploy.
References
- Feedback Capacity of the Gaussian Interference Channel to Within 2 Bits
- Tokenization and the Noiseless Channel
- Two Counterexamples to Tokenization and the Noiseless Channel
- Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations
- Tokenization Falling Short: On Subword Robustness in Large Language Models
- Retrofitting Large Language Models with Dynamic Tokenization
- The Token Tax: Systematic Bias in Multilingual Tokenization
- Interference Channels With Rate-Limited Feedback
- Tokenization as Cultural Erasure: How Corpus Composition Shapes the Representation of Aymara Morphology in NLP Systems
- IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models
- Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation