Latent Round-Trip Translation for Cross-Lingual Transfer Without Bitext

A multilingual language model fine-tuned on English task data usually works worse on other languages, and much worse on token-level tasks such as named-entity tagging and slot filling. A plausible cause is that target-language sentences land in a slightly different region of the model's internal representation space than English sentences, so the English-trained classifier head sees inputs unlike those it was trained on. We propose to treat this as an unpaired image-to-image translation problem, but carried out on hidden vectors instead of pixels. We learn a small map that turns target-language token representations into "English-looking" ones and a reverse map that does the opposite. The maps are trained only on unlabeled target-language text and labeled English data, with no parallel sentences, dictionaries, or machine translation (MT). A round-trip constraint and a label-preservation constraint keep them honest.

Computer vision has a mature recipe for exactly this situation. Its key lesson, from CyCADA, is that matching distributions and requiring round-trip reconstruction is not enough. Without an explicit constraint that task labels survive translation, the generator "flips" labels toward whatever is common in the target domain. In cross-lingual transfer the label constraint is cheap, because English gold labels are available. The proposal asks a narrow question: does this constrained, unpaired latent translation close the structured-prediction transfer gap better than simpler ways of aligning representations? The data budget is a few thousand unlabeled target sentences.

Core Claim

Simple first-moment corrections and sentence-level alignment can largely fix classification transfer. They should not fix token-level structured prediction, where language differences reshape representations in a context-dependent way rather than shifting them uniformly. We claim that a pair of low-capacity token-level maps, trained with three losses, recovers a meaningful share of the structured-prediction gap from monolingual target text alone. The first loss is distribution matching against the other language, the second is round-trip reconstruction, and the third is preservation of English gold labels through the round trip. We further claim that the label-preservation term, not the round trip by itself, carries most of the benefit, which mirrors CyCADA's ablations.

The prior work plays three roles here. CyCADA supplies the method: its combination of pixel cycle consistency, feature alignment and semantic consistency raised SVHN→MNIST accuracy from 67.1% to 90.4%. XeroAlign motivates the token-level focus. Its MSE pull between English and machine-translated sentence embeddings transfers classification well but leaves an 8–19 point slot-filling gap behind native training. Two papers constrain the design. Zero-shot Cross-lingual Transfer without Parallel Corpus shows that bitext-free transfer is achievable, but it relies on Wikipedia inter-language entity links and an elaborate voter-gated self-training loop, and links of that kind are sparse for low-resource languages. The vision literature's warning that strict cycles invite information hiding when domains carry different amounts of information tells us to keep the maps small and the cycle tolerance loose.

What Is Being Built

We start from XLM-R Large (a multilingual masked language model) fine-tuned on English task data. The encoder and the English head $f_E$ are then frozen. Let $h$ denote a contextual token vector from a fixed upper layer. We train $G$, which maps target-language token vectors toward the English region, and $F$, which maps English vectors toward the target region. Each map is a residual network with one hidden layer on top of an affine transform, so the parameter count stays near a single adapter. A small discriminator per side judges whether a token vector looks English or target-like. The objective is

$$\mathcal{L} = \mathcal{L}_{\text{adv}}(G,F) + \lambda_c \big( \lVert G(F(h_E)) - h_E \rVert + \lVert F(G(h_T)) - h_T \rVert \big) + \lambda_s \, \mathrm{CE}\big(f_E(G(F(h_E))), y_E\big) + \lambda_s \, \mathrm{CE}\big(f_T(F(h_E)), y_E\big),$$

where $h_E$ and $h_T$ are English and target token vectors, $y_E$ are English gold token labels, $f_T$ is a target-side head trained on mapped English vectors, $\mathrm{CE}$ is cross-entropy, and the $\lambda$ weights balance the terms. The two label terms play the part of CyCADA's semantic consistency loss. Because they use gold English labels, they avoid the circularity of supervising target predictions with the model's own zero-shot guesses.

The trained system can be used in two ways. Latent translate-test feeds $G(h_T)$ to the English head. Latent translate-train uses $f_T$ on raw target vectors, having trained it on synthetic target vectors produced by $F$. These are the representation-space counterparts of the two classic MT pipelines, and comparing them is part of the study.

Research Program

The work has three parts. The first sets up the method and measures it on structured prediction across typological distance, varying the unlabeled target text from 1k to 100k sentences to trace a data-efficiency curve. The second is a diagnosis of failure modes. We will measure label flipping directly by checking whether mapped English tokens keep their gold label under $f_E$ in each class. We will also measure information hiding by testing whether the cycle can still reconstruct vectors after small perturbations that destroy low-variance directions. Typologically distant, non-Latin-script languages are where XTREME reports the sharpest drops, and they are where we expect a strict cycle to break. There we compare it with a relaxed cycle in which reconstruction is penalized only beyond a margin. The third part asks whether the benefit is really about token-level structure, by comparing gains on sentence classification against gains on tagging.

Baselines

The naive reference point is plain zero-shot transfer: the English-fine-tuned XLM-R applied directly to target text. XTREME reports an overall average of 68.2 for XLM-R Large, and XTUNE's reimplementation reports 70.0. On XNLI the XLM-R zero-shot average is 79.2% (XTREME). We will rerun it ourselves with the same checkpoint and layer choice.

The strongest competitor under the same no-bitext constraint is Bilingual Task Fitting with voter-gated self-training from Zero-shot Cross-lingual Transfer without Parallel Corpus. It reports 79.11% average F1 on CoNLL NER and 85.21% accuracy on cross-lingual sentiment. It is the strongest because it also avoids bitext and MT and has the best reported structured-prediction number in that regime. We will rerun it on our encoder with the same unlabeled target text and the same hyperparameter search budget, about 20 configurations each. We will run it twice, with and without its Wikipedia entity-link step, because those links are an extra resource our method does not use. Beating it means higher average CoNLL F1 than our rerun of BTF without entity links, and matching or exceeding the with-links version on at least the distant-language subset.

As an MT-dependent reference we include ConNER, which reports 80.29 average CoNLL F1 using translation-based and dropout consistency, and XTUNE, which reports 74.9 on XTREME. Both use resources we forgo. We treat them as a ceiling that tells us what fraction of the MT-assisted gain bitext-free latent translation recovers, not as competitors we must beat.

Three ablations isolate the key ingredient. A moment-matching map shifts and rescales target vectors to the English mean and variance; it tests whether anything beyond first and second moments matters. Removing both label terms gives plain latent CycleGAN; it tests label flipping. Removing the cycle term gives adversarial feature alignment with label preservation, which is essentially CyCADA's feature branch alone.

Why not just…?

Why not just translate the training or test data with machine translation? When good MT exists it is a serious threat, and we report it as a reference. However, Modelling Latent Translations for Cross-Lingual Transfer shows that 1-best translate-test propagates errors: sampling 12 translations plus task-aware MT tuning was needed to recover up to 2.7 points, and 5.6 points on Haitian Creole. Translate-train for tagging also requires projecting labels across word alignments. The target setting is languages without reliable MT, so the relevant comparison is among bitext-free methods.

Why not just center each language's representations on a shared mean? This is the alternative most likely to match us on classification, and we do not argue it away. The moment-matching ablation settles the question. If it equals the full method on tagging as well, the cycle and label machinery is unjustified and we will report that. Our hypothesis, testable on XTREME's tagging tasks and MasakhaNER, is that a uniform shift helps classification but leaves most of the token-level gap. The basis is XeroAlign's finding that whole-sentence alignment does not repair slot filling.

Why not just self-train on pseudo-labeled target text? The non-parallel paper needed entity-linked warm-up, a multi-encoder voter ensemble and automatic thresholding to make self-training work, and removing the warm-up cost 7.55 points. ConNER also notes the confirmation bias of pseudo-labels. Self-training is complementary rather than exclusive: if the latent maps help, they should also make a better teacher. We therefore also test self-training initialized from our maps.

Why not just train the encoder adversarially against a language discriminator? That approach is the no-cycle ablation in a slightly different place. CyCADA found that each removed component cost substantially. A direct comparison on distant languages will show whether the round trip prevents the collapse that one-directional alignment permits.

Proposed Experiments

Experiments run on XTREME's structured-prediction and classification tasks across its 40 languages, CoNLL 2002/2003 NER, MultiATIS++ and MTOP slot filling, and MasakhaNER for low-resource African languages. We found no reported result on Lacuna for MasakhaNER downstream F1, so all MasakhaNER numbers will be our own reruns. Metrics are F1 for tagging, accuracy for classification, and the gap to native target training where target training data exist.

The decisive check has three parts that must hold together. The full method beats moment matching on tagging by a clear margin, larger than the variation across three random seeds. It beats rerun BTF without entity links on CoNLL and MasakhaNER. And removing the label terms produces measurable label flipping, observed as a drop in per-class label retention for mapped English tokens. If the method helps only on close European languages, or only on classification, the claim fails and the result becomes a negative finding about latent cycles under asymmetric tokenization. The data-efficiency curve then states how few unlabeled target sentences are needed before the gains appear, which is the practical number for low-resource deployment.