A multilingual language model such as XLM-R places sentences from about a hundred languages in roughly one shared vector space. That shared space is why you can fine-tune the model on English labels and then run it on Swahili or Hindi with no Swahili or Hindi training data. This setup is called zero-shot cross-lingual transfer. The weak point is fine-tuning itself. Training on a small English dataset bends the top layers toward English-specific task cues, and accuracy on distant, low-resource languages falls well below English. In vision, a cheap structural trick addresses a closely related problem. A small nonlinear network, the projection head, sits between the encoder and the loss during training and is then thrown away. It acts as a sacrificial buffer: it absorbs the loss's narrow pressures so the encoder underneath stays general. We propose to test whether this buffer protects the shared multilingual space during English fine-tuning. We will measure that protection directly, by checking how well target-language sentences stay aligned with their English translations inside the backbone, rather than inferring it from end-task accuracy alone.
The core claim is specific and falsifiable. During English fine-tuning, a wide nonlinear projector inserted between XLM-R and the task classifier will preserve more of the backbone's cross-lingual alignment than a standard head. This will show most clearly in a "discard-and-reprobe" readout: after fine-tuning, we drop the projector and train a fresh linear classifier on the frozen backbone using English data only. That readout should beat keeping the projector on typologically distant languages, while matching or trailing it on English. If the gap between kept and discarded readouts does not grow with language distance, the sacrificial-buffer story is wrong for this setting.
What already exists and what is different
The buffer idea comes from SimCLR, which applies a contrastive loss to the output of an MLP head and shows that the features before the head transfer better than the features after it. Two vision papers carry the idea into supervised training and supply the method we reuse. Revisiting the Transferability of Supervised Pretraining: an MLP Perspective inserts a discardable BatchNorm-ReLU MLP before the cross-entropy classifier and removes a final-layer transferability drop (63.1% vs 55.9% on ImageNet-21K eval-D). Revisiting Supervision for Continual Representation Learning shows the same insertion prevents forgetting under sequential tasks. Both papers study ResNets under domain or class shift; neither touches language models or language shift.
Theory papers explain why the buffer might help and also constrain the design. Investigating the Benefits of Projection Head for Representation Learning argues that deeper layers greedily upweight whichever features most easily reduce the training loss, and that the head absorbs this greed. It also reports that a fixed, non-trainable reweighting head can match a learned MLP. That result directly threatens the "nonlinearity matters" premise, so we test it rather than assume it away. Projection Head is Secretly an Information Bottleneck frames the head as limiting how much loss-specific information flows back into the encoder. It finds that structural bottlenecks such as Top-k sparse projectors beat soft regularizers on CIFAR-100 (+3.87% vs +1.26%). It also warns that a bottleneck that is too tight breaks training, which argues for a capacity sweep rather than a single setting.
On the multilingual side, three papers motivate the problem or are the closest competitors.
Better Fine-Tuning by Reducing Representational Collapse documents that XLM-R forgets general features during fine-tuning. It counters this with noise-consistency regularization (R3F) and a spectrally normalized linear head (R4F). That is a constraint on the output head's smoothness; it is not a buffer that is removed before inference.
Consistency Regularization for Cross-Lingual Fine-Tuning (XTUNE) enforces agreement across augmented views, including machine translation and code-switching. This requires bilingual resources that our method does not use.
Disentangled Contrastive Learning for Zero-Shot Multilingual Dense Retrieval is the closest existing work. It puts two linear projection heads on a multilingual encoder, one for semantics and one for language, and keeps them orthogonal. It trains on WikiMatrix parallel text and word alignments, uses the projected semantic vector directly at test time, and names nonlinear heads as future work.
Our proposal differs from that paper on every one of those axes. It uses a single nonlinear head, which is discarded at test time. It uses no parallel text or alignment tools in training. And it targets task fine-tuning rather than retrieval pretraining.
One point a reader outside the subfield should know: the usual sequence-classification head on RoBERTa-style models is already a small tanh MLP on the first token. So "put an MLP on top" is not, by itself, new. What is new here is three things. The head is treated as disposable. The zero-shot readout problem that disposal creates is solved explicitly. And the mechanism is measured inside the backbone.
Research Program
The model is XLM-R, base and large. During English fine-tuning, the backbone output $h$ passes through a projector $g$ and then a linear task layer. For token-level tasks such as named-entity recognition and extractive question answering, $g$ is applied to every token. We vary three properties of $g$. Its depth ranges over one to three layers. Its width ranges from half to four times the hidden size. Its type is either dense ReLU or Top-k sparse, the latter following the information-bottleneck paper.
Discarding the head creates a problem flagged in the research file: in zero-shot transfer there is no target-language data to train a new classifier. We compare three readouts. In the kept readout, $g$ stays in place. In the discard-and-reprobe readout, the backbone is frozen, $g$ is dropped, and a linear classifier is trained on $h$ using the same English training set. In the nearest-centroid readout, English class means of $h$ serve as prototypes; this applies only to the classification tasks. The discard-and-reprobe readout can work because a linear map on a frozen, well-aligned $h$ has little room to latch onto English-specific features that the projector already soaked up.
The mechanistic measurement uses the fact that XNLI test sets are human translations of the same English items across 15 languages. We define alignment retention for language $\ell$ as
$$R_\ell = \frac{A_\ell(\theta_{\text{ft}})}{A_\ell(\theta_{\text{pre}})},$$
where $A_\ell(\theta)$ is the top-1 accuracy of retrieving each English test sentence's translation in language $\ell$. The search runs over mean-pooled backbone features $h$ from model parameters $\theta$, either pretrained ($\theta_{\text{pre}}$) or fine-tuned ($\theta_{\text{ft}}$). These parallel sentences are used only for measurement and never for training.
The hypothesis predicts three things. First, $R_\ell$ will be higher with a wide nonlinear projector than with the standard head. Second, the increase in $R_\ell$ will be largest for languages that start out least aligned. Third, the per-language gain in accuracy will track the per-language gain in $R_\ell$. We additionally track the effective rank of $h$ on target-language inputs, to connect the multilingual setting to the dimensional-collapse findings of Jing et al..
Baselines
The naive reference point is standard fine-tuning with the default head attached directly to the backbone. This is the setting behind the XTREME numbers. On XNLI, XTREME reports 79.2% average zero-shot accuracy for XLM-R; the same paper reports an XTREME average of 68.2 for XLM-R Large. We rerun it ourselves with the same seeds, learning-rate grid and step budget as every other arm, so that gains are not artifacts of tuning.
The strongest competitor that, like ours, needs no extra resources is R3F/R4F from Better Fine-Tuning by Reducing Representational Collapse. It targets exactly the failure we hypothesize, collapse of general features during fine-tuning. The file records an 81.4 average for R3F with XLM-R on XNLI but does not state the model size, so we reproduce R3F and R4F under our protocol rather than compare to that figure across setups. We give both methods an equal tuning budget over their noise scale and our projector width. Beating R3F means a higher mean target-language XNLI accuracy over five seeds, with non-overlapping seed spreads on the distant-language subset (Swahili, Urdu, Thai, Arabic, Hindi). We also run the combination of R3F and the projector, since the two attack different sites and may add up.
The strongest resource-heavy reference is XTUNE from Consistency Regularization for Cross-Lingual Fine-Tuning, which reports an XTREME average of 74.9 against its own XLM-R Large baseline of 70.0. We do not expect to beat a method that uses machine translation and bilingual dictionaries. We report how much of its gain over vanilla fine-tuning a translation-free projector recovers.
Three ablations isolate the key ingredient. The first is a matched-parameter linear projector: a stack of linear layers with no activation, the same size as the MLP. It asks whether nonlinearity, rather than extra parameters, does the work. The second is the fixed non-trainable reweighting head from Investigating the Benefits of Projection Head for Representation Learning. The third is the kept-versus-discarded readout comparison on the identical trained model, which isolates the "sacrificial" claim from the "bigger head" claim.
Why not just…?
Why not just keep the default head, which is already a small MLP? Because the default head is never discarded, so it cannot test whether the backbone beneath it stayed general. Our kept-readout arm with a wide projector is effectively this alternative at larger size. If the kept readout matches discard-and-reprobe everywhere, and $R_\ell$ does not move, the honest conclusion is that head capacity, not sacrifice, is all that matters.
Why not just use R3F or spectral normalization? These methods are a real threat, because they already reduce collapse on XNLI. However, they work by smoothing the output function, while the projector changes where the loss's greedy pressure lands. The information-bottleneck paper found structural bottlenecks outperformed pure regularization in vision. That is suggestive but from a different domain, so the head-to-head and combined runs, together with $R_\ell$ for each, settle the question here. R4F also needs a linear classification matrix, while a projector applies unchanged to span and token heads. We check whether that matters by running MLQA and NER.
Why not just read out a middle layer, or freeze the top layers? This is the most dangerous alternative. If the problem is that the top layers lose alignment, simply probing a lower layer might recover the same gain at no cost. We therefore include a frozen-backbone linear probe at every layer of the vanilla-fine-tuned model, and a top-k-layers-frozen fine-tuning arm. The projector earns its place only if discard-and-reprobe at the final layer beats the best vanilla layer probe on distant languages.
Why not just translate the training or test data? XNLI shows translate-test is strong for high-resource pairs. But it costs a translation system at inference, and the research file notes that high-quality translation is unavailable for many low-resource languages. We report translate-train as a ceiling on the XNLI languages and focus our claims on AmericasNLI and MasakhaNER from Composable Sparse Fine-Tuning for Cross-Lingual Transfer, where translation is weakest.
Proposed Experiments
The main experiments run on XNLI, PAWS-X, WikiAnn NER and MLQA from the XTREME suite, all trained on English only. The low-resource stress test uses AmericasNLI and MasakhaNER. Metrics are accuracy for classification, F1 for NER, and F1/exact match for MLQA. For every arm we also report $R_\ell$ and effective rank. Each configuration runs five seeds, because zero-shot transfer variance across seeds is large enough to hide one-point effects.
Three results would decide the question.
The first is whether discard-and-reprobe beats the kept readout on distant languages while not beating it on English. That is the signature of a buffer absorbing English-specific pressure.
The second is whether the per-language accuracy gain over vanilla fine-tuning correlates with the per-language gain in $R_\ell$ across the 15 XNLI languages. Without that link, any accuracy gain is unexplained regularization.
The third is whether the nonlinear projector beats both the matched linear stack and the fixed reweighting head. If the fixed head ties, as it did in vision, we will report that the cross-lingual benefit comes from reweighting alone, which is a simpler and still useful finding.
A width-and-sparsity sweep tests the information-bottleneck paper's warning: transfer should peak at an intermediate capacity, and very tight Top-k projectors should hurt English accuracy first.
References
- A Simple Framework for Contrastive Learning of Visual Representations
- Revisiting the Transferability of Supervised Pretraining: an MLP Perspective
- Revisiting Supervision for Continual Representation Learning
- Investigating the Benefits of Projection Head for Representation Learning
- Projection Head is Secretly an Information Bottleneck
- Better Fine-Tuning by Reducing Representational Collapse
- Consistency Regularization for Cross-Lingual Fine-Tuning
- Disentangled Contrastive Learning for Zero-Shot Multilingual Dense Retrieval
- Understanding Dimensional Collapse in Contrastive Self-supervised Learning
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation
- XNLI: Evaluating Cross-lingual Sentence Representations
- Composable Sparse Fine-Tuning for Cross-Lingual Transfer