Fertility-Normalized Online Mixing of a Base and a Tokenizer-Swapped Language Model

When a language model is adapted to a new language, the usual first step is to replace its tokenizer. The tokenizer is the component that splits text into subword pieces, and it gets replaced with one built for the target language, so each word costs fewer pieces. This lowers fertility, the average number of tokens per word. That choice is made once, offline, and it can fail. Dobler and de Melo found that swapping the tokenizer and continuing pretraining helped Mistral-7B on Arabic but left it worse than the unadapted model on German, a language it already handled reasonably. Signal processing has a well-worn answer to this situation, where a "fast" learner helps in some regimes and hurts in others. The answer is to run a fast and a slow adaptive filter side by side on the same input and blend their outputs with one mixing weight that is itself learned online from the combined error. The blend then follows whichever filter is doing better at the moment.

We propose to do this for language models. The two models are a base model with its original, high-fertility tokenizer and a tokenizer-swapped adapted model. A single mixing weight between them is updated word by word on the text actually observed in a stream. The key difficulty, and the part that is new, is that the two models chop the same text into different numbers of pieces. Mixing and updating at the token level therefore gives the high-fertility model more votes and more correlated gradient steps per word. We argue that the mixing weight must be defined and updated at the word level and normalized for error power. Without that, the combination systematically misjudges which model is better, most severely on exactly the scripts where fertility is highest.

Core Claim

Two claims can be tested. First, a word-level mixture of a base model and a tokenizer-swapped model can be updated online with a power-normalized rule borrowed from adaptive filter combinations. On any stationary stream it will stay close to the better of the two models. On streams that switch language, domain or script, it will beat both, because it recovers quickly after each switch. Second, the naive alternative of mixing and updating per token is biased toward the higher-fertility model. The size of that bias grows with the fertility ratio between the two tokenizers. Normalizing to words removes the bias.

Mechanism and Prior Work

The combination layer comes directly from the adaptive-filter literature. Arenas-García et al., also indexed as Combinations of Adaptive Filters, show that the output $y(n)=\lambda(n)y_1(n)+(1-\lambda(n))y_2(n)$ of a fast and a slow filter can be driven toward whichever constituent currently has lower error. They also show that normalizing the mixing update by an estimate of error power (their cvx-PN-LMS) is what keeps the mixer stable when signal levels change. They further report that such combinations are more robust than single-filter variable-step-size schemes when the noise level is unknown. We reuse the parameterization and the power normalization. We do not reuse the error model: their errors are squared errors on scalar outputs, and ours are log-losses on words.

Concretely, let $w_t$ be the $t$-th word of the stream and $h_t$ the preceding text. Let $p_A(w_t\mid h_t)$ and $p_B(w_t\mid h_t)$ be the probabilities the base and adapted models assign to that word, each computed by multiplying the model's own subword probabilities across however many tokens the word takes. The mixture probability is

$$p_\lambda(w_t\mid h_t)=\lambda_t\,p_A(w_t\mid h_t)+(1-\lambda_t)\,p_B(w_t\mid h_t),\qquad \lambda_t=\sigma(a_t),$$

and the unconstrained parameter $a_t$ takes one gradient step on the observed word's log-loss, divided by a running power estimate $r_t$:

$$a_{t+1}=a_t+\frac{\mu_a}{r_t+\delta}\,\lambda_t(1-\lambda_t)\,\frac{p_A-p_B}{p_\lambda}, \qquad r_t=\beta r_{t-1}+(1-\beta)\Big(\frac{p_A-p_B}{p_\lambda}\Big)^2 .$$

Here $\sigma$ is the logistic function, $\mu_a$ is the mixer step size, $\beta$ is a forgetting factor, and $\delta$ is a small constant for numerical stability. Because the mixture is defined over words, it is a proper probability for every word regardless of how either tokenizer splits it. Because there is one update per word, the number of mixer steps does not depend on fertility.

The token-level alternative multiplies per-token mixtures and updates once per token. It lets a Gujarati word that costs 18.44 native-script tokens under an English-centric tokenizer (the figure reported by RomanSetu) trigger eighteen highly correlated mixer steps. Those steps are driven mostly by the easy, nearly deterministic continuation tokens inside the word.

The convergence-stability literature explains both why this matters and why the mixer should be safe. Mateos et al. decompose tracking error into a lag term that falls with step size and a noise term that rises with it. That decomposition is the reason a single adapted model cannot be right for both German and Arabic. Zhou et al. prove that recovery time after a shift in test-time adaptation scales as $C_\phi/(B\alpha^2\epsilon)$. In that expression, $C_\phi$ measures how strongly consecutive observations are correlated, $B$ is batch size, $\epsilon$ is the target excess risk, and $\alpha$ measures how well the unsupervised proxy loss aligns with the true loss; adaptation collapses when $\alpha$ turns negative.

Their result both motivates and constrains our design. It motivates it because splitting a word into many pieces is precisely a way of inflating $C_\phi$ per unit of information, which predicts slower and noisier recovery under token-level updating. It constrains it because it warns against proxy losses. Our mixer is therefore trained only on the log-loss of text already observed in the stream, which is the true loss. That puts us in their $\alpha=1$ regime and avoids the collapse they prove for entropy-style objectives. Their analysis and experiments cover vision classification (CIFAR-C, ImageNet-C); we carry the quantity they call recovery complexity over to text streams as an empirical measurement, not as a proven bound.

The fertility literature supplies the problem and the two constituents. The Token Tax shows on AfriMMLU that fertility explains up to 50% of variance in accuracy, and that each additional token per word costs up to 18 points. It also notes that fertility is confounded with pretraining data volume. Dobler and de Melo supply the adaptation recipe: tokenizer swapping plus continued pretraining in pure bfloat16. They also supply the German/Arabic split that makes a fallback mechanism necessary. Our proposal differs from both: they treat tokenizer choice as a fixed offline decision, while we keep both tokenizations alive and let the observed stream decide how much to trust each, word by word. The research file contains no prior work that combines language models with different tokenizers through an online filter-style mixer.

Research Program

We will build two constituent pairs by reproducing the Dobler and de Melo recipe on Mistral-7B, for German and for Arabic. We will add a third pair for Swahili so that AfriMMLU and the Swahili portion of Belebele can be evaluated. Each pair consists of the base model and a tokenizer-swapped, continued-pretrained model. For each pair we implement three things:

  1. The word-level power-normalized mixer described above.
  2. A token-level mixer that is identical except for its unit of update.
  3. A word-level mixer without power normalization.

Word boundaries come from whitespace and script-aware pre-tokenization that is shared by both models. For scripts where whitespace words are unreliable, we fall back to fixed-length byte chunks, which keeps the mixture well defined.

The analysis has three parts. First, we measure the direction and size of the token-level bias as a function of the fertility ratio between tokenizers. We do this by comparing the long-run mixing weight under token-level and word-level updating on stationary streams where the better model is known. Second, we measure recovery time after a switch point, defined as the number of words needed for the mixture's running loss to return within $\epsilon$ of the better constituent. We plot it against the fertility of the incoming segment to test the $C_\phi$ prediction.

Third, as an exploratory extension, we will test the affine combination that the filter literature uses to beat both constituents when their errors are weakly correlated, as in the tracking gains reported by Arenas-García et al. and Xuan et al.. A linear mixture of probabilities with a weight outside $[0,1]$ is not a valid distribution. We therefore test the affine idea only in log space, and only on multiple-choice scoring, where renormalization runs over just four answer options.

Baselines

The naive reference points are the two constituents used alone: base Mistral-7B with its original tokenizer, and the tokenizer-swapped adapted model. The file reports no downstream numbers for either on German or Arabic. Dobler and de Melo report only that the Arabic model outperformed baselines including AceGPT-7B, and that the German model underperformed base Mistral-7B. So there is no reported result found on Lacuna for exact scores, and we will produce both constituents ourselves under an identical evaluation harness.

The strongest competitor is oracle per-language selection: for each language, pick whichever constituent scores better on a held-out development split. This is what a careful practitioner would actually deploy. It already captures the German-versus-Arabic lesson with no runtime machinery. It is run on the same models and data, with the development split disjoint from the test split. The proposal beats it only if the online mixer does better on switching streams, or does better per item on stationary benchmarks. Merely matching it would not count.

A second strong competitor is a static mixture whose weight is grid-searched per language on the development split. It gets the same development data and a tuning budget comparable to our search over mixer step size and forgetting factor.

The key ablation is the token-level mixer, which is identical to ours except for where the update happens. Isolating fertility normalization means showing that it loses to the word-level mixer, and that its loss grows with the fertility ratio. The unnormalized word-level mixer separately isolates the contribution of power normalization.

For external context on Belebele, the Belebele paper reports 51.1% average accuracy for zero-shot GPT-3.5-Turbo and 60.2% for XLM-V under Translate-Train-All across 122 languages. These are not same-model baselines; they only calibrate how hard the benchmark is.

Why not just…?

Why not just pick the better model per language? This is the most serious threat. On clean single-language benchmarks it may match the mixer, and we will report that outcome if it happens. The mixer's case rests on two regimes where per-language selection has nothing to act on. One is streams that switch language, domain or script mid-document. The other is within-language variation, such as English named entities, numbers and loanwords inside Arabic text, where the base model may be locally better. The switching-stream experiments and the per-item analysis below decide whether the online weight adds anything beyond routing.

Why not just average the weights of the two models? Weight-space interpolation is undefined here. The two models have different vocabularies, so their embedding and output matrices do not correspond row by row. Averaging only the transformer body would still leave two incompatible input and output layers. Mixing in word-probability space is the level at which the two models actually become comparable.

Why not just romanize, as RomanSetu does? RomanSetu cuts Gujarati fertility from 18.44 to 3.39 tokens per word and gains about 8 chrF on translation. However, it depends on a lossy transliterator that introduces spelling errors when converting back to native script. It also requires continued pretraining with mixed English data, and it does not apply to Arabic-script or Latin-script targets like German. Romanization is complementary, since a romanized model could serve as the low-fertility constituent. It does not remove the need to decide online how much to trust the adapted model.

Why not just tune the continued-pretraining learning rate, or rely on pure bfloat16 as a regularizer? Both act offline and globally. Dobler and de Melo's German model was trained in pure bfloat16 and still fell below the base model. A single learning rate also faces the lag-versus-noise tradeoff for the entire deployment distribution at once. The filter literature reports that combinations are more robust than variable-step-size single learners when the noise level is unknown. We test this directly: we include a German model continued-pretrained at a reduced learning rate as an additional constituent-alone baseline.

Proposed Experiments and Decisive Checks

Streaming language modeling. We build streams by concatenating document blocks, with switch points placed every few hundred to few thousand words. Some streams are single-language control streams in German, Arabic or Swahili. Others alternate between the base model's strong language (English) and the target language. A third type consists of target-language text dense with English entities. The metric is bits per byte, which is neutral across tokenizers, together with recovery time in words after each switch.

The first decisive check has two parts. On every control stream, the word-level mixer must stay within a small margin of the better constituent: we fix 0.02 bits per byte before running. On switching streams, it must beat both constituents and the tuned static mixture. The second decisive check is that the token-level mixer shows a long-run weight bias toward the base model that increases with the fertility ratio: this should be largest for Arabic, the pair whose original tokenizer fragments the target script most, and smallest for German. We also expect the token-level mixer to show slower recovery on high-fertility segments. If neither the bias nor the recovery gap appears, the fertility-normalization claim is wrong.

Multiple-choice benchmarks. We use Belebele for German, Arabic and Swahili, and AfriMMLU for Swahili. Answers are scored by log-likelihood. The mixing weight for each item is set by running the online mixer over the item's passage and question, so the weight reflects which model explains that particular text better.

The third decisive check is whether this per-item weight beats oracle per-language selection in accuracy. If it only ties, we conclude that dynamic mixing helps under non-stationarity but adds nothing on stationary benchmarks, which is still a useful negative result. On the same items we run the log-space affine variant. It counts as a success only if it beats both constituents and the convex mixer, which would be the language analogue of the filter literature's "better than either" region.

To address the confound flagged by The Token Tax, we will report the mixer's gains separately for items grouped by base-tokenizer fertility, within each language. This reduces the extent to which fertility effects are mistaken for effects of data volume.

Cost. Running two models doubles inference compute. Adapting the filter literature's reduced-overhead idea, we will measure how often the mixing weight saturates. Where it stays near 0 or 1 for a sustained window, the inactive model can be evaluated only intermittently. We will report accuracy against compute for this gated variant alongside the full mixer.