Many new language models replace most of their attention layers with cheaper sequence layers. Attention lets every token look at every other token, so its cost grows with the square of the input length. The cheaper layers are either state-space layers, which compress everything seen so far into a fixed-size running memory, or local convolutions, which only look at a short window of neighbors. These "hybrid" models are judged almost entirely on English. Languages differ, though. Some are broken by the tokenizer into many more pieces per word, and some place a verb far from its subject. Both properties plausibly stress exactly what the cheap layers give up: unlimited memory and long reach. This proposal tests whether swapping attention for cheaper layers costs non-English languages more than English when a model is trained on an English task and applied directly to other languages, a setting called zero-shot cross-lingual transfer. If it does, we test whether choosing which attention layers to keep, based on how well the model's languages stay aligned, removes that cost.
The test bed is a controlled retrofit of a pretrained multilingual encoder, XLM-R. A retrofit gives exactly matched pure-attention and hybrid twins, which is impossible when comparing released models trained on different data. We replace attention layers in XLM-R with either a bidirectional state-space layer or a depthwise convolution. The two substitutes predict different failure patterns. If the fixed-size memory is the problem, degradation should track how many tokens each word becomes. If locality is the problem, it should track how far apart syntactically linked words sit. The study therefore separates two mechanisms that hybrid model papers currently treat as one "efficiency" choice.
Core Claim
We claim three things. First, when attention layers in a multilingual encoder are replaced by cheaper sequence layers, the loss in zero-shot accuracy is uneven across languages. Second, the per-language loss is predictable from two measurable language properties: how many subword tokens the tokenizer produces per word, and how long syntactic dependencies are in that language. Third, choosing which attention layers to keep by measuring how much each replacement disturbs cross-lingual alignment shrinks the gap between English and the hardest languages. Cross-lingual alignment here means whether a sentence and its translation still land near each other in the model's representation space. The comparison is against choosing layers by how well the substitute reproduces English behavior, at equal attention budget.
Where This Sits
The hybrid side supplies the motivation and the substitute layers. Jamba and Jamba-1.5 interleave Mamba state-space layers with attention and report up to an 8× smaller key-value cache, but evaluate on English long-context and reasoning tasks. Falcon-H1 mixes state-space and attention heads inside each layer, with the same English-centric evaluation. Neither asks whether the savings are paid for unevenly across languages, which is our question.
On the convolution side, CeiT inserts convolutions into vision Transformers. The research file attributes to that line the observation that repeated reshaping between grid and token layouts erodes throughput. That constrains us: we will measure wall-clock speed and memory rather than trust operation counts.
Du et al. ( SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction) argue that non-uniform hybrids usually forfeit pretrained weights. This is why we retrofit rather than pretrain. It is also a risk we must show we avoid.
The cross-lingual side supplies the hypotheses and the measurements. XLM-R shows that multilingual transfer is capacity-hungry ("the curse of multilinguality"). That raises the worry that compressing context into a fixed state removes capacity the low-resource languages need most. UDify finds that middle layers of a multilingual encoder carry language-universal syntax across different word orders. This motivates the idea that some attention layers matter more for transfer than others, and that English loss alone will not reveal which ones. Measuring Cross-lingual Transfer in Bytes finds that transfer is dominated by language-agnostic representations and correlates only weakly with syntactic distance (ρ = −0.147). This is a genuine counterweight to our locality hypothesis. If the convolution substitute shows no word-order effect, that paper predicted it.
The closest work at the intersection does not answer our question. HC2L combines acoustic and contextual representations for cross-lingual spoken language understanding through a contrastive objective, not by changing backbone layers. The Fast Conformer audio-visual recognizer is a multilingual convolution-attention hybrid, but for speech, with no zero-shot text transfer. Disentangled contrastive retrieval separates language-invariant from language-specific signal, but in pure Transformers. None of them measures how replacing attention changes the per-language transfer gap.
Research Program
We start from XLM-R base, 12 layers. For a budget of $k$ retained attention layers ($k \in \{12, 6, 3, 1\}$), we replace the attention sublayer in the other $12-k$ layers with one of two substitutes. The first is a bidirectional state-space layer: two Mamba-style scans, forward and backward, whose outputs are summed. The second is a depthwise 1D convolution with a kernel of width 7. Feed-forward sublayers and embeddings stay as pretrained.
Each substitute is initialized by layer-wise distillation. It learns to reproduce the replaced attention sublayer's output on unlabeled text, followed by a short masked-language-model healing phase. The distillation text comes from CC-100 in one of two mixes: English only, or language-balanced across the XNLI languages. The mix is itself an experimental factor, since an English-only retrofit may silently specialize the substitute.
The method contribution is the rule for choosing which $k$ layers keep attention. The English-fidelity rule keeps the layers whose replacement most increases distillation and masked-language-model loss on English. The alignment-preserving rule removes each layer in turn and measures the resulting drop in translation retrieval accuracy. That accuracy is the fraction of sentences whose nearest neighbor in another language's representation space is their own translation. We measure it on unlabeled parallel sentences from the XNLI development split, which is never used for testing, and greedily keep attention where the drop is largest. This uses no target-language task labels, so the setting stays zero-shot for the task.
For the analysis, we define the transfer gap for language $l$ as $\Delta_l = \text{acc}_{\text{en}} - \text{acc}_l$. We then fit how the change in this gap between the hybrid and the full-attention twin depends on language properties:
$$\Delta_l^{\text{hybrid}} - \Delta_l^{\text{full}} = \beta_0 + \beta_f f_l + \beta_d d_l + \epsilon_l$$
Here $f_l$ is tokenizer fertility, the average number of subword tokens per word in language $l$. The quantity $d_l$ is the mean dependency length in Universal Dependencies treebanks for $l$, measured in subword tokens, so it reflects both word order and fragmentation. The residual $\epsilon_l$ is everything else. The fixed-memory hypothesis predicts $\beta_f > 0$ chiefly for the state-space substitute. The locality hypothesis predicts $\beta_d > 0$ chiefly for the convolution substitute. With 15 XNLI languages plus 7 MLQA languages this is a small regression, so we report it with bootstrap intervals over test examples and seeds, and treat it as a directional test rather than a precise estimate.
We also profile efficiency. For every variant we measure throughput and peak memory at 512 subword tokens and at byte-length equivalents of 2–6× that, on the same GPU with the same kernels. This addresses the CeiT-line concern that layout changes erase savings.
Baselines
The naive reference point is the unmodified XLM-R base twin, fine-tuned on English MultiNLI or SQuAD and evaluated directly on the other languages. It sets each language's full-attention gap $\Delta_l^{\text{full}}$, which every hybrid is measured against. For scale, XLM-R reports 80.9% average XNLI accuracy. The file does not say which model size that number is for, so we will reproduce our own base-size number rather than borrow it. On MLQA, the only per-language transfer numbers in the file come from MLQA for XLM: 68.0 F1 on Spanish, 54.8 on Arabic, 48.8 on Hindi, and roughly a 40% exact-match drop from English. These give a sense of how wide the gaps already are before any compression.
The strongest competitor for the efficiency question is a homogeneous Transformer of equal inference cost, obtained by distilling XLM-R into a shallower all-attention student. It keeps the uniform architecture that optimized kernels favor, and the scaling story in XLM-R suggests uniform stacks transfer well. It receives the same distillation corpus, the same total GPU-hours, and the same hyperparameter search (learning rate and warmup over the same grid) as the hybrids. It is compared at matched measured throughput at each sequence length, not at matched parameter count.
The second strong competitor is local-window attention, where each token attends only within a fixed neighborhood. It is the attention-native way to get locality, which separates "locality hurts" from "convolution hurts." We match its window size to the convolution's kernel width and also test a 64-token window.
For the zero-shot comparison as such, translate-test is the classic strong baseline: target-language inputs are machine-translated into English and classified by an English model. XNLI found it the strongest approach across most languages, above 70%. We report it as cited and do not rerun it, since no translation system is in the file.
The key ablation is English-fidelity versus alignment-preserving layer selection at identical $k$, substitute, and distillation mix. It isolates the proposal's main ingredient.
What counts as success differs by claim. The alignment-preserving hybrid beats the English-fidelity hybrid if it reduces the mean gap increase over the five hardest languages (by full-attention gap) with non-overlapping bootstrap intervals, while keeping English accuracy within one point. The hybrid as a design beats the distilled homogeneous student if, at matched throughput, it has a lower worst-language gap. If it does not, we will report that hybrids are not worth it for multilingual encoders at these lengths.
Why not just…?
Why not just distill into a smaller uniform Transformer? This is the real threat. At 512 tokens, attention's quadratic cost is a modest share of encoder compute, so a shallower all-attention student may match the hybrid's speed with less multilingual damage. We do not argue around this. The matched-throughput comparison across sequence lengths decides it. The likely outcome is that hybrids only win past some length, and locating that crossover per language is itself a useful result, because high-fertility languages reach it sooner.
Why not just evaluate released hybrids like Jamba on XNLI? Released hybrids and released Transformers differ in data, tokenizer, and scale, so any multilingual gap could not be attributed to the layer type. The Jamba papers provide no matched pure-attention twin. A retrofit is the cheapest way to hold everything else fixed. We will check external validity by testing whether the fertility effect we find also appears in the fixed-memory regime the Jamba papers describe at long lengths.
Why not just translate the test set into English? Translate-test is strong on XNLI according to XNLI. But it answers a different question: it bypasses the model's multilingual representations entirely, and it adds a second model's latency, which defeats the efficiency motive. According to MLQA and zero-shot reading comprehension, direct transfer is preferable for span extraction because translation breaks answer boundaries. We therefore report it as a reference ceiling, not a competitor to the architectural question.
Why not just add learned layer weighting, as UDify does? UDify recovers morphology and syntax from a pure Transformer by learning a weighted mix of its layers. It is cheap, and it could also compensate for damaged layers in a hybrid. We add it on top of every variant, including the baselines. The selection-rule claim must hold with layer weighting present.
Decisive Checks
The experiments use XNLI (15 languages, accuracy) and MLQA, both same-language and mixed-language question-passage settings (F1 and exact match). Mean dependency length comes from Universal Dependencies v2.3. Croatian and Myanmar XNLI (C-XNLI, Myanmar XNLI) serve as held-out probes; no reported results were found on Lacuna for these. No reported result was found on Lacuna for retrofitted hybrid multilingual encoders on any of these benchmarks. Every hybrid number is therefore new and compared only against our own reproduced twin. Each configuration uses three fine-tuning seeds.
Four outcomes are decisive. First, if the gap increase is flat across languages for both substitutes, the "hidden tax" claim is false and hybrids can be adopted multilingually on English evidence alone. That is worth knowing. Second, if the state-space substitute's loss tracks fertility while the convolution's tracks dependency length, the two-mechanism story is confirmed. If neither tracks either property, the byte-transfer paper's language-agnostic view wins. Third, the alignment-preserving rule must beat the English-fidelity rule on the hardest languages at $k=3$ and $k=1$, where the choice matters most. Fourth, profiling must show real wall-clock and memory gains at the lengths where hybrids are recommended. Otherwise the CeiT-line concern about layout overheads stands, and the recommendation is withdrawn.
References
- Jamba: Hybrid Transformer-Mamba Language Models
- Jamba-1.5: Hybrid Transformer-Mamba Models at Scale
- Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
- Incorporating Convolution Designs into Visual Transformers
- SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction
- SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction
- Unsupervised Cross-lingual Representation Learning at Scale
- 75 Languages, 1 Model: Parsing Universal Dependencies Universally
- Measuring Cross-lingual Transfer in Bytes
- HC2L: Hybrid and Cooperative Contrastive Learning for Cross-Lingual Spoken Language Understanding
- Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer
- Disentangled Contrastive Learning for Zero-Shot Multilingual Dense Retrieval
- MLQA: Evaluating Cross-lingual Extractive Question Answering
- XNLI: Evaluating Cross-lingual Sentence Representations
- Jamba: A Hybrid Transformer-Mamba Language Model
- Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
- C-XNLI: Croatian Extension of XNLI Dataset
- Myanmar XNLI: building a dataset and exploring low-resource approaches to natural language inference with Myanmar