Learned translation metrics are neural models that read a source sentence, a machine translation, and a human reference, then predict how a person would rate the translation. They are trained almost entirely on human ratings for high-resource language pairs. When they are pointed at languages where a single word carries case, number, person and agreement in stacked affixes, such as Finnish, Kazakh, Zulu or Nahuatl, their agreement with human judges drops. On the AfriMTE challenge set, Evaluating WMT 2024 Metrics Shared Task Submissions on AfriMTE reports that the 13-billion-parameter MetricX-24 reaches a weighted Pearson correlation of 0.6269, and that all systems collapse on English–Luo and English–Twi. The languages that most need automatic evaluation almost never have the human ratings needed to fix this by ordinary fine-tuning.
Adversarial domain adaptation trains a feature extractor to fool a second network that tries to tell source data from target data, using only unlabeled target examples. It looks like a natural fix, but applied naively to a metric it should fail for a specific reason. Translations into low-resource, morphologically rich languages are on average much worse than translations into German or Czech. Forcing the two feature distributions to match therefore forces the metric to treat bad target translations as if they were average source translations. This proposal makes the adversary compare languages only among translations the metric already rates as similar in quality. It also adds a small set of automatically generated grammatical corruptions that the metric must still penalize, which guards against the adaptation making it blind to real errors.
Core Claim
Our hypothesis is that learned metrics carry a language-specific shortcut. Features such as how many subword pieces a word is split into, or how surface-divergent a hypothesis is from its reference, act as quality cues in high-resource training data. These cues misfire in agglutinative languages, where heavy fragmentation and surface divergence are normal even for good translations. Because sentence length and morphological complexity vary within a single language, the shortcut corrupts the ranking of sentences within that language, not just its overall calibration.
We claim that quality-conditional adversarial alignment removes this shortcut from unlabeled target data, where unconditional alignment cannot. Quality-conditional means the discriminator sees the metric's features together with the metric's own predicted quality bin, so alignment happens only within matched quality strata. The method should raise within-language correlation on AfriMTE and on the WMT22 morphologically rich subset with zero target human labels. A sensitivity constraint keeps it penalizing agreement and case errors. The novelty lies in where the domain discriminator sits and what it is conditioned on: it targets the quality model's shortcut under label shift, rather than "making representations language-invariant" in general.
Research Program
The metric follows the COMET recipe from COMET: A Neural Framework for MT Evaluation, which pools layer-wise encoder representations of source, hypothesis and reference and regresses a human score. We use AfroXLMR-76 as the encoder so that the comparison with AfriCOMET-1.1 isolates the training objective. Labeled source data are the WMT human direct-assessment ratings that COMET-style metrics are trained on.
Unlabeled target data are new machine translations into the target languages. We produce them with several systems of deliberately varied quality, including NLLB-200 and weaker in-house models, on FLORES-200 sentences disjoint from the AfriMTE evaluation sentences. Each translation is paired with its reference. We also report a transductive variant that uses the unlabeled evaluation inputs, kept separate.
Let $z = G(s, h, r)$ be the pooled features for source $s$, hypothesis $h$ and reference $r$. Let $f(z)$ be the predicted score, $b = \text{bin}(\text{sg}[f(z)])$ the predicted quality bin with gradients stopped, and $D(z, b)$ a discriminator predicting whether the example comes from a source language or a target language. Training minimizes
$$ \mathcal{L} = \mathcal{L}_{\text{reg}}(f(z_{\text{src}}), y) \;-\; \lambda\, \mathcal{L}_{\text{dom}}\big(D(z, b), d\big) \;+\; \mu \sum_{(h, \tilde h)} \max\big(0,\; m - f(z_h) + f(z_{\tilde h})\big), $$
Here $y$ is the human score on source data, $d$ the domain label, $\lambda$ and $\mu$ loss weights, and $m$ a margin. The pair $(h, \tilde h)$ consists of a translation and a copy with an injected agreement or case error. The minus sign on the domain loss is implemented with a gradient reversal layer, as in Yu et al.: an identity map on the forward pass that negates gradients on the backward pass.
Conditioning on $b$ is what separates this from textbook alignment. Within a quality bin, source and target examples should be indistinguishable apart from the shortcut, so the discriminator can only win by exploiting fragmentation or language identity, and that is what the extractor is pushed to discard.
Prior adversarial work supplies the method. Adversarial Domain Adaptation for Classification of Prostate Histopathology Whole-Slide Images aligns features across hospitals and pairs the alignment with a consistency constraint so the adversary cannot erase grading information. Our hinge term plays the same role for grammaticality. Adversarial Topic-Aware Prompt-Tuning for Cross-Topic Automated Essay Scoring adversarially adapts an English essay scorer across topics. It reports that minimax training is unstable for regression and stabilizes it with joint classification. We borrow that stability trick through the score-bin conditioning, but our domains are languages and our task is reference-based translation scoring, not rubric scoring.
Episodic Training for Domain Generalization trains feature extractors to work with classifiers from other domains when no target data exist. We use its leave-one-language-out idea as a competing baseline, not as our mechanism.
The corruptions come from Universal Dependencies treebanks, following the guided perturbations of Morph Call: Probing Morphosyntactic Content of Multilingual Transformers. That work also constrains the design: it finds that fine-tuning can degrade masked-token prediction by up to 10% while morphosyntactic information concentrates in the middle layers. We therefore probe middle-layer case and agreement information before and after adaptation. Corruptions exist only for languages with treebanks, such as Finnish, Estonian, Czech and Kazakh. Whether the anchor's protection transfers to languages without treebanks, such as Bribri, is itself a measured question.
The motivation from the metric literature is twofold. UTER reshapes edit costs to forgive shared roots in morphologically rich languages but has no semantic model. FUSE stacks lexical, phonetic and LaBSE features into regressors that need target human ratings. Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks shows that COMET, BLEURT and BERTScore overpenalize small meaning-preserving edits in German-to-English, and calls for adversarial training without doing it. We take up that call for morphologically rich targets, where many character-level differences are grammar rather than noise.
Baselines
The naive reference points are BLEU and chrF. On AfriMTE, Wang et al. report BLEU at a weighted Pearson of 0.2394. We also include the source-trained metric itself, which is COMET on AfroXLMR-76 trained only on WMT ratings with no adaptation. This is the "do nothing" point for our method.
The strongest existing system is MetricX-24, at 0.6269 weighted Pearson on AfriMTE as reported by Wang et al.. It is twenty times our size, so we treat it as a ceiling we aim to approach, not as a matched comparison.
The matched strongest competitor is AfriCOMET-1.1, which the same paper describes as COMET fine-tuned on AfroXLMR-76 and as performing on par with MetricX-24. No reported overall AfriCOMET-1.1 correlation was found on Lacuna, so we rerun it on our splits. It shares our encoder and the improvement it embodies is in-language pretraining, which makes it the right test of whether adversarial alignment adds anything beyond better pretraining.
Two ablations isolate the key ingredient. Unconditional alignment drops $b$ from the discriminator and tests the label-shift argument. Augmentation-only keeps the corruption hinge and the unlabeled target data but has no discriminator.
All learned variants share the same encoder, source ratings, unlabeled target pool, and a fixed budget of hyperparameter trials. The trials are selected on a held-out source language pair, never on target human ratings, because target ratings are assumed unavailable. That source pair is Czech for the African experiments and Estonian for the WMT22 experiments.
Beating the field means three things: exceeding source-only COMET and AfriCOMET-1.1 in mean within-language Pearson on AfriMTE across three seeds, beating augmentation-only by more than seed variance, and not losing ground on the corruption check. On AmericasNLP 2025, FUSE reports a development Pearson of 0.8455 and Spearman of 0.8001 (Raja & Vats) using target human ratings. We report how much of that supervised result an unlabeled method recovers, rather than claiming to beat it.
Why not just…?
Why not just pretrain the encoder on more target-language text? This is the most serious threat. AfriCOMET's results suggest that in-language pretraining recovers much of the gap at 560M parameters. Our design stacks the adversary on the same AfroXLMR-76 encoder, so a null gain over AfriCOMET-1.1 would show that the shortcut is a representation-coverage problem that pretraining already solves. In that case we would report this and drop the adversarial claim. The persistent collapse on Luo and Twi reported by Wang et al. is our reason to expect pretraining alone leaves something on the table.
Why not just normalize scores per language? Z-scoring a metric per language fixes pooled, cross-language correlation. It cannot change Pearson or Spearman computed within one language, because any increasing per-language transform leaves the within-language ranking unchanged. We therefore evaluate primarily within language and report pooled numbers with and without normalization, so any gain that normalization alone explains is visible.
Why not just train with the corruption pairs and skip the adversary? Consistency and margin training on synthetic pairs is simpler and may be enough. The augmentation-only ablation answers this directly. If it matches the full model, the adversary is unnecessary and the contribution shrinks to a morphology-aware augmentation recipe. That result is still useful, but it is not the claim made here.
Why not just use a bigger model or prompt a large language model? MetricX-24 at 13B still collapses on Luo and Twi (Wang et al.), and the AmericasNLP 2025 findings ( Findings of the AmericasNLP 2025 Shared Tasks on Machine Translation, Creation of Educational Material, and Translation Metrics for Indigenous Languages of the Americas) report that prompted language models trailed fine-tuned systems on Nahuatl. We include MetricX-24 as a ceiling but do not expect scale alone to remove a training-data shortcut.
Proposed Experiments
The primary evaluation is within-language sentence-level Pearson and Spearman correlation with human direct assessment. We measure it on AfriMTE, on the WMT22 subset covering Finnish, Estonian, Latvian, Lithuanian, Czech, Kazakh, Zulu and Xhosa as used by UTER, and on AmericasNLP 2025 Shared Task 3. No per-language correlations for the WMT22 subset were found on Lacuna, so we recompute UTER and chrF on our splits.
Three checks decide whether the core claim holds.
The first is a shortcut diagnostic. On each target language, we measure the partial correlation between the metric's error against human scores and the subword-pieces-per-word ratio, controlling for sentence length. The hypothesis predicts this correlation is substantial for source-only COMET and shrinks under conditional alignment more than under unconditional alignment or augmentation-only. If source-only COMET shows no such correlation, the mechanism we posit is absent and the project should be redirected.
The second is a label-shift check. We vary the quality mix of the unlabeled target pool, from mostly strong to mostly weak systems. Unconditional alignment should degrade as the pool gets worse while conditional alignment stays flat. This is the direct test of the reason for conditioning.
The third is a blindness check. We measure the score drop on held-out UD-derived case and agreement corruptions, both in treebank languages and in treebank-free languages through manually written corruptions. We also run middle-layer probes for case and number following Morph Call, to confirm that the adapted metric still sees grammar.
As a downstream follow-up, we use the adapted metric to rerank NLLB candidates and measure inflectional diversity of the output. Machine Translationese shows that optimizing systems toward reference-matching scores homogenizes morphology. A metric that stops penalizing legitimate morphological divergence should select less homogenized output, and if it does not, that bounds how much the adaptation changes what systems are rewarded for.
References
- Evaluating WMT 2024 Metrics Shared Task Submissions on AfriMTE (the African Challenge Set)
- COMET: A Neural Framework for MT Evaluation
- Fan-Net: Fourier-Based Adaptive Normalization for Cross-Domain Stroke Lesion Segmentation
- Adversarial Domain Adaptation for Classification of Prostate Histopathology Whole-Slide Images
- Adversarial Topic-Aware Prompt-Tuning for Cross-Topic Automated Essay Scoring
- Episodic Training for Domain Generalization
- Morph Call: Probing Morphosyntactic Content of Multilingual Transformers
- UTER: Capturing the Human Touch in Evaluating Morphologically Rich and Low-Resource Languages
- FUSE : A Ridge and Random Forest-Based Metric for Evaluating MT in Indigenous Languages
- Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks
- Findings of the AmericasNLP 2025 Shared Tasks on Machine Translation, Creation of Educational Material, and Translation Metrics for Indigenous Languages of the Americas
- Findings of the AmericasNLP 2025 Shared Tasks on Machine Translation, Creation of Educational Material, and Translation Metrics for Indigenous Languages of the Americas
- Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation