A multilingual language model fine-tuned on English labels often fails on Yoruba, Quechua, or Amharic. One reason is simple: the target-language sentences produce hidden states that sit in a different region from the English ones the task head was trained on. Most fixes either need translated or parallel text, or they adapt the language without reference to the task. We propose a third route. Unpaired image-to-image translation learned to map one image distribution onto another without matched pairs, using two tools: optimal transport (moving one cloud of points onto another at minimum total cost) and cycle consistency (requiring that a round-trip map returns the input). We apply the same tools to hidden states. We learn a small map, inserted at a middle layer of the model, that moves target-language token states onto the English token states the task model already handles well. The map is trained only on unlabeled, unpaired target and English text.
The new ingredient is where the matching cost is measured. Earlier unpaired alignment measured distance in a generic embedding space. We measure it in the coordinates of the English task model itself, meaning distance after the hidden states pass through the task-adapted upper layers. Transport effort therefore goes into the directions that change the label, and directions that only encode language identity are left alone. Our hypothesis is that this task-warped, token-level transport narrows the zero-shot gap on structured prediction, where gaps are largest, without any bitext. We also identify when a much cheaper closed-form alignment would do just as well.
Core Claim
Fine-tuning on a task warps the representation space so that labels become separable. This is the core observation of task-specific embedding adaptation, from polarity separation in Predicting Polarities of Tweets by Composing Word Embeddings with Long Short-Term Memory to the residual task shifts in Detecting Twitter posts with Adverse Drug Reactions using Convolutional Neural Networks. The warping is learned only from English. The closest analogue to our proposal is Mapping Unseen Words to Task-Trained Embedding Spaces, which learns a function that carries words absent from task training into the task-adapted space. It does so by regressing on the displacements observed for seen words, within one language and with static vectors.
We treat a whole target language as the "unseen words." We learn the carrying map without any paired displacements, using distribution matching. The claim is that this matching works when the cost is taken in task-warped coordinates and does not work as well when it is taken in generic coordinates. The task-warped choice is what separates the proposal from prior unpaired alignment. It is also the ingredient our main ablation isolates.
Relation to Prior Work
Two papers supply the method. Unsupervised Cross-lingual Transfer of Word Embedding Spaces imported CycleGAN-style dual maps and Sinkhorn distance into cross-lingual word embeddings. Sinkhorn distance is an entropy-smoothed optimal transport cost. That paper beat MUSE on English–Spanish lexicon induction (41.95% vs 33.58%). A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings (VecMap) achieved the same kind of alignment with an orthogonal map and self-learning, relying on the assumption that the two languages' spaces have the same shape. Both operate on static fastText vectors, and neither involves a downstream task. We reuse their transport and cycle machinery on contextual token states and attach it to a task model.
The research file also points to a direct tension: task fine-tuning breaks the shape-equivalence these methods rely on. We test that tension rather than assume it away.
Several papers motivate the proposal. Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs found that middle layers of Llama 3 and Qwen 2.5 are where languages are most aligned. Contrastive alignment there improved MASSIVE slot filling by +1.5 F1 (+3.8 for the lowest-resource languages). It also reported that ordinary fine-tuning is "alignment-neutral." The method needs Tatoeba parallel sentences, however, and those are unavailable for most AmericasNLI or MasakhaNER languages. XeroAlign: Zero-Shot Cross-lingual Transformer Alignment aligned sentence-level [CLS] vectors of translated pairs and still left an 8–19 point slot-filling gap to target-language training. We read this as evidence that alignment must happen at the token level.
Image translation constrains the design. Correspondence-aware translation in Patch Your Matcher and Curvilinear Structure-preserving Unpaired Cross-domain Medical Image Translation shows that local structure survives unpaired mapping only when the constraints are local. The CycleGAN literature reviewed in Generative Adversarial Networks: Recent Developments documents two failures that rule out a naive port. First, distribution matching alone is under-constrained and can shuffle content while still matching the target distribution. Second, strict cycle loss fails when the two domains carry unequal information. Unequal subword counts across languages, the tokenizer fertility problem that Franken-Adapter addresses by changing the vocabulary, are exactly such an asymmetry.
For these reasons we avoid adversarial training. We transport word-level rather than subword-level states, and we keep the map a small residual near the identity.
Research Program
We fine-tune an English task model in the standard way, with full fine-tuning for XLM-R Large and LoRA for Llama 3 8B. We pick a middle layer $\ell$. Let $\phi$ denote the frozen task-adapted layers from $\ell$ up to the layer just before the task head. We collect two clouds of word-level hidden states at layer $\ell$: $\mu_s$ from unlabeled English text and $\mu_t$ from unlabeled target text. Each word's state is the mean over its subword states, which removes the fertility mismatch.
We learn a residual map $T(h) = h + g(h)$, where $g$ is a two-layer network. We also learn an inverse $F$. The training objective is
$$\min_{T,F}\; W^{\epsilon}_{c}\big(T_\#\mu_t,\ \mu_s\big) \;+\; \lambda\,\mathbb{E}_{h\sim\mu_t}\|F(T(h))-h\|^2 \;+\; \beta\,\mathbb{E}_{h\sim\mu_t}\|g(h)\|^2,$$
The terms are as follows. $T_\#\mu_t$ is the target cloud after mapping. $W^{\epsilon}_{c}$ is the Sinkhorn distance with smoothing $\epsilon$, computed in minibatches. $\lambda$ weights the round-trip term, which keeps $T$ from discarding content. $\beta$ keeps the map close to the identity.
The task-warped cost is
$$c(h, h') = \|\phi(h) - \phi(h')\|^2,$$
so two states count as close when the English task model treats them alike. At test time, $T$ is applied to target-language states at layer $\ell$, and the rest of the model runs unchanged.
Every hyperparameter is chosen without target labels. This covers $\ell$, $\lambda$, $\beta$, and $\epsilon$. Selection uses held-out Sinkhorn distance on unlabeled target text together with English dev accuracy through $T\circ F$. Using target-dev tuning would quietly turn a zero-shot method into a few-shot one.
The analysis has three parts. First, we measure whether task fine-tuning reduces the shape-equivalence between language clouds, using VecMap-style similarity-profile agreement before and after fine-tuning. Second, we measure whether the transport preserves content. FLORES-200 parallel sentences are used for evaluation only: we check whether mapped target words land near their English translations. Third, we measure how much of the gain a closed-form Gaussian transport already captures. That map matches mean and covariance with an affine map.
Baselines
The naive reference point is plain zero-shot transfer: the English-fine-tuned model applied to target text unchanged. Published anchors exist. XTREME reports an average of 68.2 for XLM-R Large across its tasks. AmericasNLI reports zero-shot accuracy "hovering just above" the 33.3% chance level. We rerun this baseline with our own seeds and use our numbers for comparison.
The strongest competitor that uses the same data we use (English labels plus unlabeled target text) is Composable Sparse Fine-Tuning for Cross-Lingual Transfer (LT-SFT). It adds a sparse task difference learned in English to a sparse language difference learned by masked language modeling on target text. The file gives it only as gains over MAD-X: +1.8 F1 on MasakhaNER, +2.5 accuracy on UD POS, +3.7 LAS on parsing, and +1.9 accuracy on AmericasNLI. MAD-X stacks separately trained language and task adapters. No absolute LT-SFT score is reported in the file, so we reproduce both LT-SFT and MAD-X ourselves.
To make the comparison fair, every method gets the same unlabeled target text, matched in tokens. It also gets the same English labeled data, the same backbone, and the same number of GPU-hours for adaptation, with the LT-SFT discovery run included in its budget. Each method receives the same label-free hyperparameter search budget. Beating LT-SFT means a mean gain across languages that exceeds the seed-to-seed spread on MasakhaNER and UD POS. Results are reported per language, so the gain cannot come from one outlier such as Quechua.
Consistency Regularization for Cross-Lingual Fine-Tuning (XTUNE) is the strongest output-level method. It reports 74.9 average on XTREME against its XLM-R Large baseline of 70.0. Its augmentations include machine translation, so we run it in two forms: with translation, and with only subword sampling and noise to match our data conditions.
Two reference methods use more supervision than we do and serve as ceilings. Translate-Train reaches 49.12% average accuracy on AmericasNLI. Middle-layer contrastive alignment with Tatoeba bitext gives +1.5 F1 on MASSIVE. If we come close to these without paired data, that is a meaningful result. Falling short of them would not falsify the proposal.
Three ablations isolate the key ingredient: generic cost, where $\phi$ is the identity so the cost is computed in raw layer-$\ell$ coordinates; no round-trip term ($\lambda=0$); and subword-level instead of word-level transport. The proposal stands or falls on the first ablation. If task-warped cost does not beat generic cost, the connection to task-specific adaptation is decorative.
Why not just…?
Why not just match the mean and covariance of the two languages in closed form? This is the most serious threat. Mean-shifting toward English is the first moment of exactly the transport we learn. If an affine Gaussian map recovers most of the gain, the nonlinear map and the cycle loss are not worth their cost. We run Gaussian transport in both generic and task-warped coordinates. The two outcomes would be read differently. If closed-form transport wins on sentence classification but loses on token-level NER and POS, then nonlinearity matters exactly where XeroAlign's 8–19 point gap lives. If it wins everywhere, we report that as the finding and recommend the closed-form task-warped variant.
Why not just mine pseudo-parallel sentences with the model's own embeddings and run middle-layer contrastive alignment? This also needs no human bitext. It is a real threat for languages the base model already represents reasonably well. Mining depends on retrieval quality, which the file shows is weak even for high-resource pairs: Tatoeba accuracy is 57.3% for XLM-R in the XTREME paper. It should be weaker still for AmericasNLI languages, which are absent from pretraining. We implement this baseline with the same text budget and plot each method's gain against the language's retrieval accuracy. Our prediction is that mining matches us at high retrieval accuracy and falls behind at low retrieval accuracy.
Why not just continue masked-language pretraining on target text? Language-adaptive pretraining gives large gains in some cases. Quechua reached nearly 62% on AmericasNLI, and Amharic NER improved by over 60% in MasakhaNER. It also fails in others, such as Otomí, and the Middle-Layer paper shows it is alignment-neutral. The approaches are complementary, so we test the combination: continued pretraining first, then transport. The question is whether transport adds gains on top of continued pretraining. Combined with the MAD-X and LT-SFT runs, this also covers the modular-adapter alternative.
Why not just Translate-Train? Where a decent translator exists, Translate-Train is the stronger option, and we say so; it is a ceiling in our comparison, not a competitor. Our target regime is languages without usable translation systems. Zero-shot Reading Comprehension by Cross-lingual Transfer Learning also reports that translation pivots can hurt by introducing errors, which we check on XQuAD.
Proposed Experiments
The main suite combines several benchmarks. MasakhaNER covers ten African languages. UD v2.7 POS covers the typologically distant XTREME languages. WikiANN NER from XTREME tests breadth, and MASSIVE slot filling allows comparison with the Middle-Layer results on Llama 3 8B. These four are structured tasks. As a contrast, we add XNLI and AmericasNLI, which are sentence classification tasks. The prediction is that transport helps structured tasks more than sentence classification, because output-level methods already handle sentence classification reasonably well. Metrics are entity-level F1, POS accuracy, slot F1, and accuracy, averaged over five seeds and reported per language.
Three checks are decisive. The first is the task-warped versus generic cost ablation on structured tasks, which must show a gain larger than seed variance. The second is the Gaussian closed-form comparison described above. The third is the content-preservation probe on FLORES-200. Gains that coincide with poor word-level correspondence would mean $T$ has learned to move states toward high-confidence English labels rather than toward their meanings. That is the text analogue of CycleGAN's horse that keeps its texture but changes its pose, and we would count it as a failure even if F1 rose.
A final analysis relates gains to tokenizer fertility and to the measured loss of shape-equivalence after fine-tuning. These two analyses test whether word-level pooling and task-warped cost address the specific failure modes the file identifies.
References
- Predicting Polarities of Tweets by Composing Word Embeddings with Long Short-Term Memory
- Detecting Twitter posts with Adverse Drug Reactions using Convolutional Neural Networks
- Mapping Unseen Words to Task-Trained Embedding Spaces
- Unsupervised Cross-lingual Transfer of Word Embedding Spaces
- A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
- Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
- XeroAlign: Zero-shot cross-lingual transformer alignment
- Patch Your Matcher: Correspondence-Aware Image-to-Image Translation Unlocks Cross-Modal Matching via Single-Modality Priors
- Curvilinear structure-preserving unpaired cross-domain medical image translation
- Generative Adversarial Networks: Recent Developments
- Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation
- AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages
- Composable Sparse Fine-Tuning for Cross-Lingual Transfer
- MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
- Consistency Regularization for Cross-Lingual Fine-Tuning
- MasakhaNER: Named Entity Recognition for African Languages
- Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model