Fencing Off Language Identity: Cartesian-Factored Hidden States for Modular Multilingual Models

Multilingual translation and generation models often produce fluent text in the wrong language. Ask a model trained mostly on English-centric pairs to translate Turkish into Gujarati, or to summarize a Swahili article in Swahili after being fine-tuned only on English, and it frequently answers in English or in the source language. A likely cause is that the model stores "which language am I writing" in the same hidden dimensions as "what am I saying." Any update to the shared weights can then nudge language identity off course. Verification of large physical systems faced a similar problem and solved it with what is called Cartesian factoring. The expensive geometric reasoning is done in a few small blocks of the state, while the full high-dimensional dynamics are computed as usual. Interactions between blocks are treated as a bounded disturbance.

We propose to build this structure directly into a modular multilingual model. A small, dedicated block of the hidden state carries language identity, and only language-specific adapters may write to it. Leakage from shared layers into the block is held under an explicit, cheaply computed bound. Because the block is low-dimensional, it becomes tractable to compute the set of language states the model can reach, layer by layer and decoding step by decoding step, and to check that this set stays inside the region of the target language. Three things follow from one mechanism. Language consistency is protected without the hand-chosen layer freezing current modular models rely on. Adapter routing can be done from inside the model instead of by an external language identifier. Every input receives a computed language-safety margin.

Core Claim

Our claim is that language identity can be confined to a block of $k \approx 8$–$16$ coordinates of the residual stream, the running hidden vector that every Transformer layer adds its output to, with bounded cross-talk from shared computation. This confinement does three things. First, it matches the language consistency of decoder freezing while allowing all shared layers to adapt. Second, it gives a routing signal that outperforms an external language identifier on realistic, mislabeled, or code-switched input. Third, it yields a per-input reachability margin that predicts off-target generation before decoding finishes.

The new ingredient is the coupling budget. Existing modular models add language-specific parameters but never control how much shared parameters can move language identity. We make that quantity explicit, bound it, and train against it.

Where this sits in prior work

The method comes from reachability analysis. Reach Set Approximation through Decomposition with Low-dimensional Sets and High-dimensional Matrices separates cheap high-dimensional linear algebra from expensive set geometry. It propagates dynamics in full dimension but performs set operations in two-dimensional blocks, and it reports analyzing a 10,913-variable circuit in under 7 minutes with error generally within 22% of the undecomposed analysis. Probabilities Are Not Enough decouples a ten-dimensional building model by treating heat exchange between rooms as a bounded disturbance input, which makes the problem scale linearly. Reach-Avoid-Stabilize Using Admissible Control Sets supplies the target notion: trajectories must reach a goal set while avoiding an unsafe set.

We borrow three things from these papers: the split between full-dimension computation and low-dimension geometry, the disturbance treatment of coupling, and the reach-avoid property. We do not attempt to verify an entire Transformer. Neural-network reachability tools such as POLAR-Express and Reach-SDP target small controller networks, and the AINNCS competition report shows the scale at which such tools currently run. That scale constrains us: whole-network certificates are not realistic here. We therefore restrict the set reasoning to the language block by construction.

The motivation comes from the modular multilingual literature, which has found the failure but only patched it. mmT5 adds language-specific bottleneck adapters during pre-training. It raises zero-shot target-language consistency on XL-Sum from 7% (dense mT5) to 99%, but only when the decoder feed-forward weights are frozen during fine-tuning to stop shared weights from drifting. We replace that freezing heuristic with a measured bound.

Adapting to Non-Centered Languages and Disentangling Pretrained Representation add a central-language-aware layer, which mixes a shared and a language-specific feed-forward block. It lifts Europarl "triangle" zero-shot BLEU from 1.97 to 20.56 and NusaX jav→ban BLEU from 7.1 to 15.5. These designs separate parameters, but nothing prevents the shared path from rewriting language identity. We separate the hidden coordinates themselves and bound the leakage.

The Obscure Limitation of Modular Multilingual Language Models is the most direct challenge to modular designs. It shows that MAD-X falls from 81.27% to 74.90% on MASSIVE when the oracle language label is replaced by a FastText language identifier, while dense XLM-R reaches 84.65% with no identifier at all. We take this as the central result a modular proposal must answer, and our routing experiment is designed around it.

Mechanism

Write the residual stream at layer $\ell$ as $h_\ell = [\,l_\ell;\, s_\ell\,]$. Here $l_\ell \in \mathbb{R}^k$ is the language block and $s_\ell$ is the remaining semantic block.

Language adapters write into both blocks. Shared attention and feed-forward sublayers are reparameterized so that the component of their output projection landing in the language block, $P_L W^{\text{out}}_\ell$, is penalized. Here $P_L$ selects the first $k$ coordinates and $W^{\text{out}}_\ell$ is the sublayer's output matrix. With pre-layer-normalization, a shared sublayer's input has bounded norm, so its leakage into the language block obeys

$$ \|w_\ell\| \le \|P_L W^{\text{out}}_\ell\|_{op}\, \|z_\ell\| \le \epsilon_\ell , $$

where $w_\ell$ is the cross-talk vector, $z_\ell$ is the sublayer's pre-projection activation, and $\epsilon_\ell$ is the per-layer coupling budget. This bound is sound for every input and costs a single spectral-norm estimate per layer.

The language block then evolves as

$$ l_{\ell+1} = l_\ell + a_\ell(l_\ell;\lambda) + w_\ell, \qquad \|w_\ell\|\le\epsilon_\ell , $$

where $a_\ell$ is the language-$\lambda$ adapter's write into the block. Because $k$ is small, the reachable set of $l$ across layers can be propagated with intervals or zonotopes, treating $w_\ell$ as a bounded disturbance in the manner of Badings et al. In the decoder, this propagation is repeated at each generation step. The target region for language $\lambda$ is defined by a linear probe on $l$. The avoid regions are the probe regions of all other languages. The per-input margin is the smallest distance between the reachable set and any avoid region.

Training adds two terms to the usual objective. The first is a penalty pushing $\sum_\ell \epsilon_\ell$ below a budget. The second is a probe loss that keeps languages separable within $l$. The probe loss doubles as the routing signal: the encoder's language block at an early layer selects the adapter, so no external identifier is needed.

Research Program

We build the model at mT5-Small and mT5-Base scale. Three variants are pre-trained on identical multilingual data and token budgets: dense mT5, mmT5, and the factored model. The first question is empirical: does a $k$-dimensional block actually absorb language identity? We test this by probing, and by checking that the semantic block alone gives near-chance language-ID accuracy while preserving cross-lingual retrieval and task quality.

The second question is whether the coupling budget can replace freezing. We fine-tune with all decoder weights trainable and sweep the budget. For each setting we record language consistency, task quality, and the measured $\epsilon$.

The third question concerns routing. The adapter chosen from inside the model is compared against FastText routing and oracle routing, using clean, mislabeled, and code-switched input.

The fourth question is diagnostic: does the reachability margin computed after the first few decoding steps predict an off-target final output?

Baselines

The naive reference point is dense mT5 fine-tuned without any language-specific modules. mmT5 reports that it reaches only 7% target-language consistency on XL-Sum zero-shot. Its role is to show the size of the problem, not to be beaten narrowly.

The strongest competitor on generation is mmT5 with decoder feed-forward freezing, at 99% XL-Sum zero-shot consistency as reported in that paper. It is strongest because it already solves the headline symptom with a simple recipe. A fair comparison retrains it at our scale on the same data, steps, and fine-tuning hyperparameter search budget, with adapters of the same total size. Since 99% leaves almost no consistency headroom, beating it means one of two things. Either we match consistency within one point while improving summary quality (no reported ROUGE result found on Lacuna for mmT5 on XL-Sum, so we measure both), or we stay above 95% consistency under fine-tuning schedules where freezing is removed or applied to the wrong layers and mmT5's consistency drops.

The strongest competitor on routing is MAD-X with FastText identification. The Obscure Limitation reports it at 74.90% average MASSIVE accuracy on its 24-language subset, against 81.27% with oracle labels and 84.65% for dense XLM-R. We evaluate on the same subset and splits. Beating it means exceeding 74.90% with internal routing, and recovering most of the gap to 81.27% counts as a strong result. Matching dense XLM-R is not claimed in advance.

For zero-shot translation, the reference points are LAVS from On the Off-Target Problem, which reduces average off-target rate from 29% to 8% across 90 FLORES-101 and WMT directions, and the central-language-aware layer on NusaX jav→ban, which reaches 15.5 BLEU and an 11.25% off-target rate. We reproduce LAVS on our backbone rather than comparing across models.

The ablations that isolate the key ingredient are three. The first reserves the language block but drops the coupling budget, so the shared layers can write anywhere. The second applies the coupling budget without a dedicated block, as a plain orthogonality penalty against a probe direction learned after training. The third routes from the full hidden state instead of the block. If the first ablation performs as well as the full method, the budget is not doing the work and the reachability framing is decorative.

Why not just…?

Why not just freeze the decoder feed-forward layers, as mmT5 does? This is the real threat. On standard XL-Sum zero-shot it already reaches 99%. Our case rests on three things freezing cannot provide: adaptation of the frozen layers when the downstream task needs it, a quantity that tells a practitioner which schedule is safe without trial and error, and a per-input margin. The schedule-sweep experiment settles the question. If freezing stays at roughly 99% across all reasonable schedules and tasks, and our quality gains are within noise, we will report that the factoring is useful mainly as a diagnostic, not as a training method.

Why not just fix off-target output at inference time with contrastive decoding or vocabulary splitting? Source- and language-contrastive decoding cuts translations with chrF2 below 10 by 67–83% and needs no retraining, and LAVS cuts the off-target rate to 8%. Both act on output tokens and leave the internal entanglement in place. Neither helps routing, and neither addresses the drift caused by fine-tuning. They are complementary to our method, so we run each alone and combined with the factored model on FLORES-101. If the factored model adds nothing on top of contrastive decoding, the translation claim fails.

Why not just use a dense model and skip routing altogether? Dense XLM-R's 84.65% on MASSIVE shows that for classification this may simply be the right choice, and we do not claim otherwise. For generation, the dense mT5 consistency of 7% and the triangle-setup BLEU of 1.97 indicate that dense models fail in exactly the zero-shot regime modularity targets. Our routing experiment asks the narrower question of whether modular models can stop paying the 7-point identifier penalty.

Why not just train a better language identifier, or a soft router on the full hidden state? The full-state router ablation answers this directly. The specific advantage of the block is that its low dimension allows the router's decision region to be checked against bounded input perturbations, such as code-switched spans. A full-state router permits no such check. If the full-state router is equally accurate and equally robust on code-switched input, the block offers no routing benefit.

Proposed Experiments

The decisive checks are four. The first is XL-Sum zero-shot target-language consistency and summary quality across a sweep of fine-tuning schedules, comparing the factored model with retrained mmT5 and dense mT5. The second is MASSIVE accuracy on the 24-language subset under oracle, FastText, and internal routing, plus a code-switched variant built by splicing spans across languages. The third is FLORES-101 zero-shot off-target rate and spBLEU across the 90 directions used by Chen et al., with and without LAVS and contrastive decoding, together with NusaX jav→ban as a low-resource check against the reported 15.5 BLEU and 11.25% off-target rate. The fourth is the predictive value of the reachability margin: we compute the area under the ROC curve for flagging off-target outputs from the margin after a fixed number of decoding steps, compared against the probe's raw confidence at the same step.

Two outcomes would falsify the proposal. The first is a block of reasonable size failing to capture language identity, meaning the semantic block still predicts language well. The second is the coupling budget giving no gain over the reserved-block-only ablation. Either would show that the Cartesian structure does not carry over from physical systems to learned representations.