Letting the Transport Plan Grade the Bitext: Label-Free Dynamic Weighting for Cross-Lingual Optimal Transport Distillation

Search engines for languages like Swahili or Somali can now be trained without any relevance labels in those languages. An English retrieval model, the "teacher," is trained on abundant English data. A multilingual "student" is then taught to place a Swahili query where the teacher places its English translation. The two sides are matched with optimal transport, a method that computes the cheapest way to move one set of token vectors onto another. The weak point is the parallel text that supplies those translation pairs. It is scraped from the web, and a large share of it is misaligned, partially translated, or simply about something else. Current methods treat every pair as equally trustworthy. The student is therefore pulled, with full force, toward teacher representations of sentences that do not mean what the Swahili side means.

This note proposes that the distillation itself can tell good pairs from bad ones, with no target-language labels and no clean validation set. The signal is how a pair's transport cost changes over training. If the pair is a genuine translation, its cost falls as the student learns, even when the sentence is hard. If the pair is misaligned, its cost stays high because no consistent mapping exists. We turn that cost trajectory into a smooth, per-pair weight on the distillation loss, introduced gradually after a warm-up. We test whether this weight recovers most of the gap between training on noisy web text and training on clean text. Low-resource cross-lingual retrieval is the main testbed.

Core Claim

Correctness-aware distillation methods all need ground truth to judge the teacher. Mentor-Critic Distillation scales the loss by whether the teacher's top prediction is right (Correctness-Aware Knowledge Distillation for Enhanced Student Learning). Confidence-Aware Multi-Teacher KD weights teachers by their cross-entropy against labels (Confidence-Aware Multi-Teacher Knowledge Distillation). Adaptive Instance Distillation uses $w=\exp(-\alpha L^T_{task})$, which needs the teacher's labeled task loss (Adaptive Instance Distillation for Object Detection in Autonomous Driving). Weighted Distillation with Unlabeled Examples removes the per-sample labels, but it still fits a k-NN noise model on roughly 2,000 clean validation examples (Weighted Distillation with Unlabeled Examples). That clean set is exactly what a Somali retrieval project lacks.

In bitext distillation, the unreliable component is usually the pair rather than the teacher. The English teacher is competent on English; the question is whether its input actually corresponds to the student's input. Our claim is that the transport cost's learning curve diagnoses this. Pairs whose cost stays flat while the rest of the corpus improves are very likely misaligned. Down-weighting them smoothly, at both the sentence and token level, should beat three alternatives: uniform optimal-transport distillation, teacher-entropy weighting, and static pre-filtering.

Mechanism

Let $C_i(t)$ be the transport cost between the student's token embeddings of target sentence $i$ and the teacher's token embeddings of its English side at training step $t$. We use the IPOT solver from OPTICAL (Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation). For each pair we keep an exponential moving average $\bar C_i$ of this cost and record its value $C_i(t_0)$ at the end of a short uniform warm-up. The pair's relative improvement is

$$r_i(t) = \frac{C_i(t_0) - \bar C_i(t)}{C_i(t_0)},$$

and its weight is

$$w_i(t) = \sigma\!\left(\frac{r_i(t) - \mathrm{med}_t(r)}{\mathrm{mad}_t(r)}\right).$$

Here $\sigma$ is the logistic function, and $\mathrm{med}_t$ and $\mathrm{mad}_t$ are the median and median absolute deviation of $r$ over a running buffer of recent pairs. Normalizing by the corpus's own statistics replaces the hand-tuned $\alpha$ that AID and MCD need. That tuning burden is the known sensitivity of these methods. The total weight is annealed from uniform to $w_i$ over training.

Using improvement rather than raw cost matters. A raw-cost weight, AID-style, would permanently starve hard but correct pairs, such as long sentences or rare vocabulary. Those pairs start expensive but do improve. The same idea is applied within a pair. Teacher tokens that receive transport mass only from tokens far away in the student's space (for example, an English clause with no counterpart on the target side) have their contribution to the cost reduced in proportion to that mass. This handles partial translations, which a sentence-level filter cannot.

Relation to Prior Work

OPTICAL and Sinkhorn-based summarization distillation (Improving Neural Cross-Lingual Abstractive Summarization via Employing Optimal Transport Distance for Knowledge Distillation) supply the method. Both apply optimal-transport distillation uniformly across pairs. We keep their loss and add only the weighting. Language Branch Knowledge Distillation for reading comprehension weights multiple teachers by their output entropy to suppress translation noise (Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation). That paper motivates the problem, since it shows translation noise hurts cross-lingual distillation. However, its signal looks only at the teacher's own confidence, and a confident English teacher on a mistranslated pair is exactly the failure we target.

ADVL zeroes the loss when the teacher is less confident than the student (ADVL: Adaptive Distillation for Vision-Language Tasks). That is a relative but hard gate; ours is relative to the corpus and continuous. Reinforced Iterative KD selects target-language instances with a policy network (Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognition). It pays for reinforcement-learning training and makes binary choices; we add no learned components. Mutually-paced KD down-weights noisy entity alignments through attention in temporal knowledge graphs (Mutually-paced Knowledge Distillation for Cross-lingual Temporal Knowledge Graph Reasoning). It is the closest in spirit, since it also lets training progress clean the alignments, and it reports degrading only 3.7% under 20% injected noise against more than 10% for competitors. Its mechanism, though, is tied to graph entities and is not a weighting on a transport loss over text.

Research Program

We first reproduce OPTICAL: an English teacher trained on MS MARCO, CCAligned bitext for Swahili, Somali, Tagalog and Marathi, and evaluation on CLEF 2000–2003 by mean average precision (MAP). We then add the trajectory weight, the token-level mass correction, and the annealing schedule.

The analysis has two parts. The first is controlled corruption. Into a CCAligned sample we inject 10%, 20% and 40% corrupted pairs. Corruption types are shuffled partners, truncated target sides, and same-topic non-translations. We then measure how well $w_i$ separates corrupted from intact pairs, reported as the area under the ROC curve (AUROC). We also measure whether intact pairs in the hardest cost quintile keep their weight. The second part is a 500-pair bilingual annotation per language, which checks that the weight tracks natural web noise and not only our synthetic noise. As a transfer check, we apply the same weight to the Sinkhorn summarization student on Wikilingua En2Ar and Zh2En, scored with ROUGE-L. This tests whether the signal is specific to retrieval.

Baselines

The do-nothing reference is the translate-test pipeline, which machine-translates queries into English and runs the English retriever. Next is uniform OPTICAL itself, the strongest existing method. OPTICAL reports a 13.7% MAP improvement over translate-test on Swahili and Somali, and beats it on Swahili with as few as 100,000 parallel sentences (Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation). No absolute CLEF MAP values for OPTICAL were found in the file, so we will report our reproduction's absolute numbers next to the paper's relative gain. We run it with identical bitext, student, IPOT settings, steps, and hyperparameter search budget.

Beating OPTICAL means a statistically significant MAP gain on at least three of the four languages on natural CCAligned data. Under injected noise, it also means a degradation slope at most half of uniform OPTICAL's.

The remaining baselines and ablations fill in the comparison. A static pre-filter discards the lowest-similarity pairs, scored by the student's initial multilingual encoder, at the same effective data budget. Teacher-entropy weighting follows the reading-comprehension branch method. An ADVL-style hard relative gate is included. Weighted Distillation's k-NN reweighting is given 2,000 hand-verified clean pairs; this is an upper reference that uses supervision we deliberately forgo. Three ablations isolate the key ingredient: a weight from raw current cost (the AID analogue) instead of cost improvement, a binary threshold on $r_i$ instead of the logistic weight, and sentence-level weighting without the token-level correction. All runs share one compute budget, and every tuned baseline gets the same number of search trials.

Why not just…?

Why not just filter the bitext once with sentence-embedding similarity? This is the real threat. If a pre-filter matches our method, the proposal reduces to data cleaning. Our argument is that one static score cannot separate "hard but correct" pairs from "wrong" pairs, because both score low before training. It also cannot catch partial translations inside otherwise similar pairs. The decisive test is the controlled-corruption study. We compare the AUROC of the pre-filter against the trajectory weight. We also compare how much of the hardest intact quintile each method keeps. If the pre-filter equals us on both and on downstream MAP, we will report that the cheaper method suffices.

Why not just weight by teacher confidence, as the branch-distillation work does? On a mistranslated pair the English teacher sees fluent English and is confident. Its entropy therefore carries no information about alignment. Running the entropy baseline on the injected-noise sets measures this directly. We expect its AUROC for detecting corruption to sit near chance.

Why not just use more bitext or a bigger student? The consistency study shows that scaling multilingual models raises per-language accuracy but barely improves cross-lingual consistency (Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models). More web bitext also brings proportionally more noise. We include a data-scaling curve for uniform OPTICAL at the four corruption levels. If doubling the clean-equivalent data closes the gap more cheaply than weighting, that answers the question against us.

Why not just translate at test time? Translate-test adds translation latency at every query, and OPTICAL already outperforms it by the margin cited above. It stays in the study as the reference point, not as a contender.

Decisive Checks

The idea succeeds if three things hold on CLEF. Trajectory weighting beats uniform OPTICAL and the static filter under natural noise. It shows a much flatter degradation curve than uniform OPTICAL as injected noise rises. And the raw-cost ablation loses to it specifically on the hardest intact pairs.

The idea fails in either of two cases. One is that the static filter matches it, which makes it a data-cleaning result. The other is that the weight's AUROC on annotated natural noise is near chance even when it works on synthetic noise. That would mean real web noise is not "unlearnable" in the way the mechanism assumes. The Wikilingua transfer run determines whether any success is a property of transport-based distillation in general or only of retrieval.