Multilingual language models are supposed to place a sentence and its translation near each other in their internal vector space. In practice their hidden layers do double duty. They encode what a sentence means, and they also track how a particular language says it: its word order, its script, its inflections. The best-known fixes act at only one point in the network. They either scrub language or syntax out of the final sentence embedding with an adversary, or they hand the model's last layer a learned "language ID" vector so that the network no longer has to carry that information itself. Evidence from vision suggests that a single point of intervention is not enough. When an unwanted signal is suppressed at the input or the output, it reappears in the middle layers.
This note proposes to measure whether the same reappearance happens inside multilingual Transformers, and to fix it with a specific kind of deep supervision. Deep supervision here means attaching small extra prediction heads to intermediate layers. Each head we attach is not told to erase language form. Instead it is given that form for free: a language embedding, a script embedding, and a small parser-free "form channel" that tracks word positions. The head then has to predict masked tokens from the intermediate layer. The central bet is that giving form an alternative route at several depths keeps the shared trunk language-neutral throughout, without the capacity loss that stronger intermediate constraints are known to cause. The heads are discarded at inference, so the deployed model is an ordinary encoder.
Core Claim
There are two ways to keep form out of the trunk: penalize it (adversarial scrubbing) or relieve the trunk of the need to encode it (offloading). Our claim is that offloading, applied at several depths, removes language and word-order information from the trunk's middle layers. Terminal offloading and terminal scrubbing leave those layers contaminated, and multi-depth scrubbing cleans them only at a cost to masked-language-modeling quality and to boundary-sensitive tasks. If the claim holds, we should see three things. Layer-wise language-identity probes on the trunk should be flatter and lower across all layers. Masked-language-modeling loss should be essentially unchanged. Zero-shot transfer should improve on tail languages and on extractive question answering, where syntax leakage causes answer-boundary errors.
Concretely, let $h_\ell$ be the trunk's hidden states at layer $\ell$ for a chosen set of layers $S$. Let $e_{\text{lang}}$ and $e_{\text{script}}$ be learned embeddings for the input's language and script. Let $f_\ell$ be the output of a small form channel, a two-layer network over position and subword-boundary features that is trained to predict each token's word position. The training objective is
$$ \mathcal{L} = \mathcal{L}_{\text{MLM}}\big(g_L(h_L + e_{\text{lang}} + e_{\text{script}})\big) + \sum_{\ell \in S} w_\ell \, \mathcal{L}_{\text{MLM}}\big(g_\ell(h_\ell + e_{\text{lang}} + e_{\text{script}} + f_\ell)\big) + \beta \sum_{\ell \in S} \mathcal{L}_{\text{pos}}(f_\ell). $$
Here $g_\ell$ is a lightweight prediction head ("tail") at layer $\ell$, $L$ is the final layer, $w_\ell$ are per-layer weights that are annealed toward zero late in training, $\mathcal{L}_{\text{pos}}$ is the word-position loss, and $\beta$ is its weight. The tails, the embeddings and the form channels are all dropped after training.
What Exists and Where This Sits
The proposal combines two techniques. The mechanism comes from LangSAMP, which adds language and script embeddings at the output of the final layer, just before the masked-language-modeling head, and reports better tail-language accuracy on Taxi1500. It intervenes only at the end of the network and models language as a single global category, with no notion of word order. The multi-depth structure comes from deep-supervision work in vision and signal processing. T-MLP attaches a prediction tail to every hidden layer of a coordinate network, so that shallow layers carry coarse structure and deep layers carry fine detail. Stacked Hourglass Networks supervise every intermediate module. The parser-free word-position target comes from the word position loss in VGVAE, which pushes syntax into a separate latent variable without gold parse trees. We use the same loss to give the form channel something to carry, which avoids the dependence on external parsers that ParaBART and S2DM have.
Two results motivate the proposal. Targeted Activation Penalties shows in convolutional networks that unwanted signals suppressed through input gradients reappear in intermediate activations, and that intervening directly on those activations works better. That paper suppresses the signal, it relies on spatial masks, and it leaves Transformers open. We transfer the diagnosis to token representations and swap suppression for offloading. S2DM shows that language-specific syntax leaking into representations causes answer-boundary errors in zero-shot reading comprehension. It separates meaning from syntax only in latent variables computed after the encoder.
Two results constrain the design. Tree-Planted Transformers supervise attention heads with syntactic distances and find that spreading this supervision across several layers or heads hurts language-modeling perplexity; supervising a single final-layer head worked best. When Intermediate Supervision Doesn't Help reports that intermediate targets can over-constrain hidden layers and conflict with the final loss.
Our answer to both constraints is a design choice, not an argument. Every intermediate head uses the same masked-token target as the final layer, so no layer is pushed toward a foreign target. The only extra signal is information that is handed to the head, never a penalty on the trunk. The weights $w_\ell$ are annealed. If this choice is wrong, the experiments will show a perplexity cost, and we treat that cost as a primary outcome.
Research Program
The first stage is diagnostic. We take a pretrained multilingual encoder (XLM-R base) and continue pretraining it on a fixed-token subset of Glot500-c, the 610-language corpus used by LangSAMP, under four regimes: plain masked-language modeling, LangSAMP-style terminal offloading, a terminal adversarial language discriminator with gradient reversal in the style of Kim et al., and a terminal syntax-bag discriminator in the style of ParaBART. For every layer of each resulting model we measure three things. Linear-probe accuracy for language identity and script identity. Linear-probe accuracy for word position and, on Universal Dependencies languages only, part-of-speech tags. Layer-wise retrieval accuracy on Tatoeba-style parallel sentences.
"Reappearance" has a precise meaning here: probe accuracy that is low at the intervention point but returns at earlier or intermediate layers relative to a schedule-matched control. This stage is publishable even if the method fails. It answers the open question of whether terminal disentanglement in Transformers leaves the middle of the network contaminated.
The second stage builds the multi-depth offloading model described above. We sweep the attachment set $S$ over three settings: every fourth layer, the middle third of the network, and only the layer identified by the stage-one probes as most contaminated. We also sweep whether $w_\ell$ is annealed. This stage directly tests the tension between dense supervision, as in vision, and single-point supervision, as Tree-Planted Transformers recommend.
The third stage asks whether information has been moved or destroyed. We probe the form channels $f_\ell$ and the language embeddings for word order and language identity. A successful model should show that word-order information is present in the side channel and absent from the trunk. This distinguishes offloading from the one-way suppression of Targeted Activation Penalties. It also gives a direct check on the worry that scrubbing erases meaning that correlates with syntax, such as which element a sentence puts in topic position.
Baselines
The naive reference point is the same encoder continued-pretrained with plain masked-language modeling on exactly the same token budget, data mixture and schedule. Any gain from our method must exceed what extra pretraining alone provides.
The strongest existing method on the multilingual-classification axis is LangSAMP, terminal language and script offloading. It is the nearest relative of the proposal, and it is the one thing that the "multi-depth" part must beat. The reported number is tail-language accuracy on Taxi1500 rising from 47.1% to 50.8% (LangSAMP). That result came from a different training setup, so we will re-run LangSAMP inside our pipeline with identical data, steps and hyperparameter search budget. Beating it means a tail-language Taxi1500 gain over our re-run LangSAMP that exceeds the seed-to-seed spread across three seeds, with no loss on high-resource languages.
On the reading-comprehension axis, the strongest competitor is S2DM, which reports +4.1 average exact match over XLM-100 on TyDi QA GoldP and cuts the BiPaR syntactic-boundary violation rate from 23.15% for mBERT to 6.60% (S2DM). We will fine-tune all encoders on English SQuAD-style data with identical recipes. We compare against S2DM's reported numbers and, where its code permits, a re-run on our backbone. Beating it means matching or beating its boundary-violation rate without its dependency-tree losses or a separate disentanglement module at inference.
Terminal adversarial disentanglement is the representative of the scrubbing family. It consists of a gradient-reversal language discriminator at the final layer, plus a syntax-bag discriminator in the ParaBART style. It receives the same tuning budget, including a sweep over the strength of the reversal.
Two ablations isolate the key ingredient. The first, deep supervision without offloading, attaches intermediate masked-token tails at the same layers but gives them no language embedding, script embedding or form channel. It separates the generic optimization benefit of deep supervision from the offloading effect we claim. The second, multi-depth scrubbing, places gradient-reversal language discriminators at the same layers. This is the obvious idea at this intersection, and our hypothesis predicts it will flatten the probes at a measurable cost to masked-language-modeling loss and boundary accuracy.
The post-hoc removal of dominant language directions from Yang et al. is a training-free baseline on retrieval. No reported result was found on Lacuna for it on Tatoeba or SR-B, so we run it ourselves on every encoder above.
Why not just…?
Why not just remove the language direction after training? Projecting out language-specific directions, as Yang et al. do, is cheap, and it may well match our method on sentence retrieval, where a few global directions dominate. That is a real threat on retrieval benchmarks. The projection acts on pooled sentence vectors, however, and it does nothing about word-order leakage in token-level representations. That leakage is what drives the boundary errors S2DM documents. We apply the projection to every model. If it matches us on SR-B and Tatoeba but not on TyDi QA and BiPaR, then the contribution is specifically about token-level form, and we will say so.
Why not just use LangSAMP's terminal offloading? LangSAMP is the special case of our method in which $S$ is empty. Whether the extra depths help is exactly what stage one measures. If terminal offloading already leaves the middle layers free of language-identity information, the multi-depth version is unnecessary, and the probe curves will show it before any downstream evaluation.
Why not just put adversarial discriminators at every layer? This is the most natural combination of the two directions. We expect it to fail for reasons the file already documents: dense structural constraints on Transformer internals degrade language modeling (Tree-Planted Transformers), and conflicting intermediate targets hurt the final task (When Intermediate Supervision Doesn't Help). Scrubbing also destroys word-order information that extractive question answering needs. This is not left as an argument. It is run as an ablation with a matched tuning budget, and if it wins, the offloading claim is wrong.
Why not just supervise a single well-chosen layer? Tree-Planted Transformers found single-point supervision best for syntax. The stage-two sweep includes a setting in which only the most contaminated layer is supervised. If that matches the dense settings, we will recommend it, since it is cheaper and resolves the dense-versus-single tension in favor of single-point supervision for language.
Decisive Checks
The first decisive check is the stage-one probe curves. If no terminal method shows reappearance of language identity in intermediate layers relative to the plain masked-language-modeling control, the motivation for multi-depth intervention collapses, and the project reduces to the diagnostic study.
The second is cost. Masked-language-modeling loss on held-out Glot500-c data for the offloading model must stay within the seed spread of the plain control, while multi-depth scrubbing is expected to exceed it.
The third is downstream transfer: tail-language Taxi1500 and SIB200 accuracy, SR-B and Tatoeba retrieval, TyDi QA GoldP exact match, and the BiPaR boundary-violation rate. On these the offloading model must beat both the deep-supervision-without-offloading ablation and re-run LangSAMP. A win only over the first ablation would mean that the language and form signals, not the extra heads, account for the gain. A win over LangSAMP would mean that depth matters.
The fourth is the side-channel probe. Word position should be decodable from $f_\ell$ and not from $h_\ell$. That would confirm that form was moved out of the trunk rather than deleted.
References
- LangSAMP: Language-Script Aware Multilingual Pretraining
- T-MLP: Tailed Multi-Layer Perceptron for Level-of-Detail Signal Representation
- Stacked Hourglass Networks for Human Pose Estimation
- A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations
- Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models
- Targeted Activation Penalties Help CNNs Ignore Spurious Signals
- Learning Disentangled Semantic Representations for Zero-Shot Cross-Lingual Transfer in Multilingual Machine Reading Comprehension
- Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision
- When Intermediate Supervision Doesn’t Help: Evidence from Recurrent CNNs
- Cross-Lingual Transfer Learning for POS Tagging without Cross-Lingual Resources
- A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations