Language-Blind Low-Rank Adapters for Cross-Lingual Transfer

When a multilingual language model is fine-tuned on an English task, its performance in other languages often gets worse, not better. The model has learned to map sentences from different languages into a partly shared internal space, and English-only training bends that space toward English. We propose a small, testable change to low-rank adaptation (LoRA), the common method that fine-tunes a large model by learning a small, low-rank correction to each weight matrix while keeping the original weights frozen. The change is to make the correction unable to "see" which language it is reading. We estimate, from a few thousand unlabeled sentences per language, the directions in each layer's hidden states that mainly encode language identity. We then force the adapter's input projection to ignore those directions for the whole of training. The adapter can still learn the task, but whatever it learns is applied identically to an English sentence and to its Swahili counterpart, because the only part of the input it can read is the part the languages share.

In two sentences for a colleague: English fine-tuning damages other languages because the update learns to respond to English-specific features. If we project those language-identity directions out of what the LoRA update can read, the update becomes the same function for every language, and zero-shot transfer should improve without parallel data, machine translation, or per-model layer searches.

Core Claim

The claim is that most of the cross-lingual damage from single-language parameter-efficient fine-tuning comes from the update reading language-specific components of the hidden state. A hard, persistent constraint that removes those components from the update's input should recover a large part of the transfer gap. It should do so on every layer, without having to decide in advance which layers to leave untouched.

Concretely, a LoRA layer computes $h = W_0 x + BAx$, where $W_0$ is the frozen pretrained weight, $x$ is the layer input, and $A$ (rank $r$ by input width) and $B$ (output width by $r$) are the trained low-rank factors. For each layer we collect hidden states from unlabeled text in a set of languages $L$. We then form the between-language scatter matrix

$$S = \sum_{l \in L} n_l (\mu_l - \mu)(\mu_l - \mu)^\top ,$$

where $\mu_l$ is the mean hidden state for language $l$, $n_l$ is its number of tokens, and $\mu$ is the overall mean. We add the top directions of the covariance of within-language residuals that best predict language identity. The top $k$ eigenvectors of this combined matrix form the columns of $U$, the layer's language-identity subspace. During training we reparameterize the input factor so that it can never read that subspace:

$$A \leftarrow A\,(I - UU^\top).$$

This projection is applied at every step, not only at initialization. If two inputs $x_{en}$ and $x_{sw}$ differ only along $U$, the update adds exactly the same vector to both. The frozen $W_0$ still carries all language-specific processing, including producing target-language tokens in late layers. Only the learned correction is language-blind.

Relation to Prior Work

Three findings motivate the proposal. High-Dimensional Interlingual Representations of Large Language Models shows that multilingual models align languages in intermediate layers and that English GSM8K fine-tuning erodes this alignment. Freezing the first 12 layers of Llama-3.1-8B lifts cross-lingual MGSM from 17.7% to 23.1%. Bridging the English-Arabic Medical Knowledge Gap finds, through causal patching, that Arabic failures are a routing problem in middle-to-late layers rather than missing knowledge. Exploring Alignment in Shared Cross-lingual Spaces reports that task fine-tuning can improve alignment even for languages absent from fine-tuning, so alignment damage is not inevitable. Together these suggest the problem lies in which features the update reads.

Three lines of work supply or constrain the method. SC-LoRA initializes LoRA factors from a difference of task and preservation covariances. It was tested only in monolingual English, and its authors note that parameters can drift out of the chosen subspace during training. We borrow its idea of a data-derived subspace but change both the target and the mechanism. We target the between-language directions specifically, rather than general preservation data, and we enforce the constraint throughout training, rather than only at initialization. Orthogonal Language and Task Adapters penalizes similarity between an adapter's output and the base hidden state on mBERT, reaching 71.43% XNLI average against 70.70% for MAD-X. Its constraint is about redundancy with the base model, not about language identity. Cross-Lingual Transfer with Target Language-Ready Task Adapters makes task adapters language-robust by cycling them through several trained language adapters. We seek the same invariance with a single projection, and we need no per-language adapters.

The closest competitors on the same problem take a different route. FLARE fuses English and target representations inside the LoRA bottleneck and beats standard LoRA on TyDiQA by 4.9 points on Llama 3.1. It needs a parallel English input, or an NLLB translation encoder, at every step, and it is bilingual. TLoRA (Bridging the English-Arabic Medical Knowledge Gap) adds an alignment loss to an English anchor at a layer chosen by causal patching for each model. Layer-swapping (The Unreasonable Effectiveness of Model Merging) stitches full fine-tuned models together at heuristic layer cutoffs. It also reports that freezing layers before training underperforms training all layers and reverting some. That result constrains our design: it argues against hard freezing and in favor of letting every layer adapt under a constraint. Our method uses only monolingual unlabeled text, touches all layers, and is agnostic to the number of languages. One risk is that the language subspace estimated for a low-resource language may be noisy. We test this directly below.

Research Program

The build is small. For each target model we run a forward pass over 2,000 to 10,000 sentences per language from monolingual corpora covering the evaluation languages. We compute per-layer language subspaces, pick $k$ per layer by the fraction of between-language scatter explained, and train standard LoRA with the projected input factor. The analysis has three parts.

The first part asks whether the constraint preserves the shared space. We measure Interlingual Local Overlap from the interlingual-representations paper, and CALIGN from Exploring Alignment in Shared Cross-lingual Spaces, before and after English fine-tuning, for each method.

The second part asks whether preserved alignment turns into accuracy. We compare zero-shot target-language task scores.

The third part asks where the subspace comes from and how far it generalizes. We estimate $U$ from a subset of languages, then evaluate on languages whose text was never used to estimate it. This tells us whether the language-identity directions are largely shared across languages, which the two-tier core-and-fragments picture of High-Dimensional Interlingual Representations leaves open. It also tells us whether the method helps the fragmented low-resource languages that need it most.

Baselines

The naive reference points are the base model with no fine-tuning and vanilla LoRA on all layers at the same rank, trained on the same English task data. High-Dimensional Interlingual Representations reports 17.7% cross-lingual MGSM for Llama-3.1-8B after standard English fine-tuning. We reproduce this configuration so our vanilla numbers sit on the same scale.

The strongest structural baseline is selective freezing: no updates in the first 12 layers, with the remaining layers adapted. The same paper reports 23.1% on MGSM for this setup with full fine-tuning of upper layers. We run it in two versions, as published and with LoRA restricted to the unfrozen layers. The LoRA version isolates the effect of the constraint from the effect of the layer choice. Our method beats this baseline if it exceeds both versions on MGSM averaged over non-English languages, across three seeds, by more than the seed standard deviation. The target is to exceed 23.1% in the published setting.

The strongest method-level baseline on NLU is FLARE (Language Fusion for Parameter-Efficient Cross-lingual Transfer), which reports +4.9 points over standard LoRA on TyDiQA for Llama 3.1 and +2.2 for Gemma 2. The file lists no absolute per-language scores (no reported result found on Lacuna for FLARE's absolute TyDiQA or XNLI numbers). We will therefore rerun FLARE and our method on the same backbones, data, rank, and step budget. Beating it means matching or exceeding its gain over LoRA on TyDiQA and XNLI without any parallel or translated input. Coming within one point of it while using none would already be a useful result, because FLARE's inputs are unavailable for languages without machine translation.

We also run SC-LoRA with multilingual unlabeled text as its preservation set, with $\beta$ tuned over the same grid budget as our $k$. This is the nearest subspace competitor. As a TLoRA-style reference, we run LoRA plus a mean-squared alignment loss to English on machine-translated pairs, following XeroAlign. For XNLI we report XTREME's 79.2% XLM-R figure only as context, since it comes from a different model family.

Two ablations isolate the key ingredient. The first projects out a random subspace of the same dimension $k$; if that helps as much, the gain is plain regularization and not language blindness. The second applies the projection only at initialization, SC-LoRA style; if that matches the persistent version, the claim about training drift is wrong.

All methods share the same English training data, trainable-parameter count (within 5%), learning-rate grid, and number of tuning runs.

Why not just…?

Why not just freeze the early layers, or put LoRA only on the upper ones? This is the real threat. The gain from 17.7% to 23.1% in High-Dimensional Interlingual Representations shows that layer choice alone does a lot. Against it, The Unreasonable Effectiveness of Model Merging found that freezing before training underperforms, and TLoRA located failures in middle-to-late layers that selective freezing leaves unconstrained. The layer-restricted-LoRA baseline settles the question. If it matches our method, we will report that the language projection adds nothing beyond layer selection. We also test the combination, because the two may be complementary.

Why not just use SC-LoRA with multilingual preservation data? SC-LoRA preserves world knowledge well in English (22.73 against 18.30 for vanilla LoRA). However, it protects the high-variance directions of the preservation data, which mostly carry content rather than language identity, and it constrains only the starting point. The initialization-only ablation and the head-to-head run test both differences directly.

Why not just translate? XNLI found translate-test strongest on most pairs, so we include translate-test with NLLB wherever it covers the language. It fails where machine translation is poor or absent: XNLI shows translation-based accuracy tracks translation quality, and TyDi QA documents translationese effects. We therefore expect our advantage to appear on AmericasNLI and NusaX rather than on high-resource XNLI languages. If translate-test wins everywhere, including there, that is a negative result worth publishing.

Why not an alignment loss on translated pairs? XeroAlign and TLoRA show that such losses work, but they need parallel data and, for TLoRA, per-model layer probing. Our method is the parallel-free alternative. The comparison shows how much of the gain from an alignment loss survives without parallel data.

Proposed Experiments

We use Llama-3.1-8B and Gemma 2 as backbones.

For reasoning, we train on English GSM8K and evaluate on MGSM, with extra attention to the Bengali, Swahili, and Telugu subset used in The Unreasonable Effectiveness of Model Merging. That paper reports 19.0% base, 20.4% mixed fine-tuning, and 21.5% layer-swapping on this subset.

For understanding, we train on English data and evaluate zero-shot on XNLI, TyDiQA-GoldP, and NusaX, and on AmericasNLI for languages with minimal pretraining coverage. Cross-Lingual Transfer with Target Language-Ready Task Adapters reports gains of 5–6 points over MAD-X on AmericasNLI. On every model, before and after fine-tuning, we also report Interlingual Local Overlap and CALIGN.

Four checks decide the outcome. First, the language-blind update must beat the random-subspace ablation on the non-English average; otherwise the mechanism is wrong. Second, it must preserve Interlingual Local Overlap better than vanilla LoRA, and the preservation must correlate across languages with the accuracy gain; otherwise the explanation is wrong. Third, gains must hold on languages excluded from subspace estimation; otherwise the method needs target text, and we will say so. Fourth, English accuracy must stay within one point of vanilla LoRA; otherwise the projection is removing task-relevant information. A sweep over $k$ from 0 up to the rank of the between-language scatter traces the trade-off between the second and fourth checks. That sweep is the main figure of the paper.