A multilingual language model such as XLM-R can be fine-tuned on English labels only and then used on Swahili, Hindi or Arabic text. This setting is called zero-shot cross-lingual transfer. It works because the model's middle layers map different languages into a roughly shared space. It degrades because fine-tuning sends the English task's error signal down through every layer, including the shared ones. We propose to change where each layer gets its training signal from. A second copy of the task head sits at a middle layer and is the only source of gradient for the layers below it. The usual head at the top trains only the layers above that point, and its gradient is stopped at the middle layer.
The motivation is a reinterpretation of the vanishing-gradient problem, the decay of useful error signal as it is backpropagated through many layers. In a residual Transformer with normalization, fine-tuning gradients do not literally vanish. The question here is what they carry by the time they reach the lower layers. Relay Backpropagation (Shen et al.) argued that gradients lose task-relevant information over long backward paths even when their magnitude is preserved. It fixed this with intermediate classifiers whose gradients are confined to local windows of layers. We ask a cross-lingual version of that question: as the English task gradient travels down, does its language-shared part shrink while its English-specific part grows? If so, a short, relayed path from a middle-layer head should update the shared layers in a way that transfers better.
Core Claim
Our central, testable claim is that the gradient a lower or middle layer receives from the final head becomes increasingly English-specific with backward distance. The same gradient taken from a head attached at the language-shared middle layer stays more language-agnostic. We will measure this by cross-lingual gradient agreement. For a parallel pair of inputs (an English XNLI or XQuAD example and its human translation), we compute the per-layer gradient of the task loss on each and take their cosine similarity. Gradient agreement is used only as an analysis tool. Parallel data is never used for training, so the method stays strictly zero-shot.
The method follows from the claim. The first ingredient is a task head at a middle layer $m$, trained on the same English labels as the final head. The second is a stop-gradient at layer $m$, so layers $1..m$ are updated only by the middle head and layers $m+1..L$ only by the final head. An overlapping variant lets the final-head gradient pass a few layers below $m$, as in Relay BP's windows. The total loss is $\mathcal{L} = \mathcal{L}_{\text{final}}(\text{sg}(h_m)) + \lambda\,\mathcal{L}_{\text{mid}}(h_m)$. Here $h_m$ is the hidden state at layer $m$, $\text{sg}$ blocks gradients, and $\lambda$ weights the middle head. At inference we use the final head, or optionally average the two heads' outputs.
Relation to Prior Work
Three lines of work supply the ingredients. The first is deep supervision: Fu et al. supervise every upsampling stage of a segmentation network, Nachmani et al. attach losses to every unrolled layer of a neural code decoder, and Newell et al. supervise each stage of a pose network. All three show that intermediate losses keep early layers learning. None of them concerns language, and none restricts which layers each loss may reach, which is the feature we need. The second is gradient locality: Relay BP (Shen et al.) is the direct method ancestor, with local backward windows for auxiliary classifiers. It was demonstrated only on convolutional networks for ImageNet and Places2. Late Prompt Tuning shows the same logic in English Transformers: prompts inserted at layer 13 of 24 beat input-layer prompts by 12.4 points on average. It freezes the lower half entirely and never tests transfer across languages.
The third line motivates the choice of layer. Middle-Layer Representation Alignment finds that middle layers of Llama 3 and Qwen 2.5 are the most language-agnostic. It adds a contrastive alignment loss there, but that loss needs parallel sentences, and it neither relays gradients nor measures them. Uncovering the Latent Potential of Deep Intermediate Representations shows that intermediate embeddings selected after training help underrepresented languages, a +10.9% relative gain for Urdu. Its selection method does not change how the model is trained. Sequential Reptile is the closest gradient-based multilingual method. It aligns gradient directions between languages or tasks at the output loss, a conflict between parallel training signals. Our concern is different: how the signal changes as it travels down the depth of the network. The two ideas could in principle be combined.
One paper constrains the design. When Intermediate Supervision Doesn't Help shows that step-by-step intermediate targets made recurrent CNNs collapse to at most 0.1% accuracy under a topological shift, mazes with loops. Moving from English to typologically distant languages is exactly this kind of structural shift. We take the warning seriously in two ways. First, our intermediate target is the final task label, not a prescribed procedure for reaching it. Second, we report results separately for the most distant languages, so that any brittleness shows up rather than being averaged away.
Research Program
The work proceeds in three steps.
Diagnosis on unmodified fine-tuning. We fine-tune XLM-R Base and Large on English XNLI and SQuAD in the standard way. Across training we record per-layer gradient norm and cross-lingual gradient agreement, using the parallel XNLI test sets and XQuAD. We also track per-layer Tatoeba retrieval accuracy as a measure of how well the pre-trained alignment survives. The claim is falsified at this step if agreement does not fall with backward distance from the final head, or if it is no higher for a middle-layer head. In that case we stop and report the negative finding.
Choice of the middle layer. We select $m$ from each model's per-layer retrieval profile before fine-tuning, not by tuning on target-language dev sets. We then sweep $m$ and the overlap width as a sensitivity analysis.
Full comparison. We train the relayed model and every baseline across the XTREME task suite (XTREME).
Baselines
The naive reference is standard zero-shot fine-tuning of XLM-R with a single final head, run with the XTREME recipe. The XTREME paper reports a 68.2 overall average and 79.2% XNLI accuracy for XLM-R Large. XTUNE reports its own XLM-R Large baseline at 70.0. We re-run this baseline under our budget rather than copying either number.
The ablation that isolates our key ingredient is plain deep supervision: the same middle head with gradients from both heads flowing everywhere. This is the obvious combination of the three directions. If it matches the relayed version, the stop-gradient adds nothing and the claim about gradient content is wrong. A second ablation moves the only head to layer $m$ and discards the upper layers, which tests whether the benefit is simply "use the middle layer." A third runs the relayed model without the middle head, which amounts to freezing layers $1..m$ with a top head.
Several methods are the strongest existing competitors. In the strict zero-shot setting, the strongest is Sequential Reptile. It reports 78.5 average F1 on TyDiQA against 76.5 for base multi-task learning, in its own multilingual training setup, and a gain of nearly 4 points over multi-task learning in English-SQuAD-to-MLQA transfer. We run its public recipe in the English-only setting. On benchmarks where it has no reported number we write "no reported result found on Lacuna."
The top reported XTREME number is XTUNE at 74.9 average. XTUNE uses machine-translated and code-switched data, so it is not strictly zero-shot. We report it as an upper reference and also test relayed supervision stacked on top of XTUNE. Layer-wise learning-rate decay is a cheap competitor tuned with the same budget. For it we found no reported result on Lacuna for cross-lingual transfer. We also include selecting the best intermediate embedding after training, in the spirit of LOES.
Every method gets the same English training data and the same number of optimizer steps. Each gets an equal hyperparameter budget of about 20 trials, selected on the English dev set only, and each is run with three seeds. We count relayed supervision as beating a baseline if two conditions hold. Its average over the structured-prediction and question-answering tasks must exceed the baseline by more than the spread across seeds, with at least a 1-point average gain over naive fine-tuning. And it must not lose on the classification tasks.
Why not just…?
Why not just freeze the lower layers during English fine-tuning? This is the most serious threat. If the only problem is that English gradients overwrite shared layers, freezing solves it at no cost. The evidence against freezing is indirect. Sequential Reptile and XTUNE both benefit from updating the whole network, and token-level tasks such as NER and POS likely need adapted lower-layer features. Our design sits between freezing and full fine-tuning: lower layers still learn, but only from a short, language-shared path. The "no middle head" ablation is exactly this frozen baseline. If it matches relayed supervision, we will say so, and the contribution shrinks to the gradient diagnosis.
Why not just add auxiliary heads at intermediate layers, without any stop-gradient? That is standard deep supervision, and it could plausibly work by keeping middle layers task-aware. Our hypothesis predicts it is worse. It still lets the English-specific final-head gradient reach the shared layers, and it adds more of it. The gradient-agreement measurements and the head-to-head ablation decide this directly.
Why not just read the task off a middle layer, or pick the best layer after training? The Middle-Layer Alignment results suggest that passive use of middle layers leaves alignment weak: retrieval accuracy was 39.4% before active optimization and 73.2% after. Discarding the upper layers also throws away capacity that span selection may need. The truncated-head and after-training-selection baselines test this.
Why not just translate the training data? Translation methods are strong where translation systems exist. XNLI reports translate-test within 3 to 6 points of English. Translation is unavailable for many low-resource languages, and Zero-shot Reading Comprehension found that it sometimes hurt compared with direct transfer. We treat translation-based methods as complementary, which is why we test relayed supervision stacked on XTUNE.
Proposed Experiments and Decisive Checks
The benchmarks are XNLI, PAWS-X, WikiAnn NER, UD POS, XQuAD, MLQA and TyDiQA-GoldP from XTREME, all trained on English only. For reference, MLQA reports XLM transfer of 68.0 F1 in Spanish and 48.8 F1 in Hindi. We expect the largest gains where the transfer gap is largest: token-level tasks, and languages with non-Latin scripts or rich morphology.
The first decisive check is mechanistic. Gains per language should correlate with the per-language improvement in gradient agreement and with how well Tatoeba retrieval accuracy is preserved after fine-tuning. A gain without these signatures would mean the method helps for a reason other than the one we claim.
The second check is the brittleness test motivated by the maze results. We examine the most distant languages in TyDiQA and WikiAnn separately. If relayed supervision helps near languages such as Spanish but hurts distant ones, intermediate supervision is imposing English structure, and the approach fails in the regime that matters most.
The third check is robustness. The result should hold across choices of $m$ near the retrieval peak, and on both XLM-R Base and Large. That rules out a single tuned layer as the explanation.
References
- Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networks
- Stacked Deconvolutional Network for Semantic Segmentation
- Learning to decode linear codes using deep learning
- Stacked Hourglass Networks for Human Pose Estimation
- Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts
- Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
- Uncovering the Latent Potential of Deep Intermediate Representations
- Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning
- When Intermediate Supervision Doesn’t Help: Evidence from Recurrent CNNs
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation
- Consistency Regularization for Cross-Lingual Fine-Tuning
- XNLI: Evaluating Cross-lingual Sentence Representations
- Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
- MLQA: Evaluating Cross-lingual Extractive Question Answering