Measuring Metric Bias Against a Second Instrument: Label-Free Correction of System-Dependent Error in Translation Metrics

Automatic translation metrics are used as rulers: we score every system with the same metric and rank them. Recent work shows the ruler stretches depending on what it measures. A learned metric such as XCOMET can agree well with human judges sentence by sentence, yet it systematically overrates some translation systems and underrates others, enough to flip their rankings. When a new system arrives, nobody knows which way the ruler will bend for it without paying for fresh human annotation. This note proposes estimating that bias without human labels. The idea is to read it off the disagreement between the target metric and a second metric built on a different principle, in the same way a Kelvin probe measures a surface's work function against a probe tip that has been calibrated separately.

The core claim is falsifiable. For a new translation system, the gap between two metrics from different families, once both are mapped onto the human rating scale, predicts how far each metric is off for that system. A small, strongly regularized correction based on this gap should cut system-dependent error on held-out systems without any new human labels. If the claim holds, metric reports for unseen systems could carry an estimated per-system bias. If it fails, we learn that metric biases are correlated across families, which bears directly on the common practice of reporting several metrics together.

Where the idea comes from

Two literatures in the file meet here, with unequal weight. The machine translation (MT) evaluation side supplies the problem. A Measure of the System Dependence of Automated Metrics fits a human-rating mapping $f_k$ for each system and a single global mapping $f_G$. It defines a per-system Expected Deviation $\mathrm{ED}_k$, the average gap between the human rating predicted by $f_G$ and the one predicted by $f_k$, and summarizes the spread as SysDep $=\max_k \mathrm{ED}_k-\min_k \mathrm{ED}_k$. On WMT23 Chinese→English, XCOMET reaches a segment-level correlation of 0.65 but a SysDep of 2.816 human-rating units, and it inverts the ranking of Lan-BridgeMT. The paper measures the problem but offers no correction. Its per-system mapping $f_k$ also needs human labels for system $k$, so it cannot help with a system that has none. That is the gap we target.

The device-physics side supplies the measurement design, not a mechanism. In Minimizing Contact Resistance and Flicker Noise in Micro Graphene Hall Sensors Using Persistent Carbene Modified Gold Electrodes, the gold electrode's work function (roughly, how strongly a material holds its electrons) is measured as a contact potential difference against a probe. The authors then shift that work function, from 5.5 to 4.2 eV with a molecular surface layer, to match graphene. Three lessons carry over. First, the useful quantity is a relative offset between two materials, not an absolute value. Second, the reference probe must be calibrated independently of the thing it measures. Third, there is a narrow window: 10 mM of the surface molecule aligns the interface, while 15 mM adds disorder and makes the mismatch worse, so over-correction is a real failure mode. Each lesson maps onto a design choice below. The analogy plays no other role in the proposal.

Core Claim and Mechanism

Let $m$ be the target metric (XCOMET), and let $a$ be an anchor metric from a different family, such as a structured LLM judge or a character n-gram score. Let $g_m$ and $g_a$ be global isotonic maps from each metric's score to the human scale, fitted only on training systems. For a held-out system $k$ with source segments $S_k$, define the label-free disagreement

$$D_k^{m,a}=\frac{1}{|S_k|}\sum_{s\in S_k}\big(g_m(m(s))-g_a(a(s))\big),$$

which is computable from system outputs alone. If the per-system errors of $m$ and $a$ are not perfectly correlated, $D_k$ carries information about $\mathrm{ED}_k^m$. We fit a low-dimensional linear predictor $\widehat{\mathrm{ED}}_k=\beta^\top D_k$ across training systems and language pairs, where $D_k$ stacks the disagreements with two or three anchors. We then report the corrected system score $g_m(m)-\lambda\,\widehat{\mathrm{ED}}_k$. The shrinkage factor $\lambda\in[0,1]$, chosen by nested cross-validation, is the "concentration window": the physics result warns that full correction can overshoot.

Independence of the anchor is a hard constraint, the counterpart of calibrating the probe tip separately. AI-Assisted Human Evaluation of Machine Translation shows that when one evaluator is primed by another, the resulting judgments cannot validate the primer. By the same logic, an anchor trained on the same MQM data as XCOMET may share its biases. We therefore treat "shared training data" as an explicit ablation axis.

Research Program

The work has three stages. The first is a measurement study. For every WMT23 metrics-task system in en-de, he-en and zh-en, we compute the true $\mathrm{ED}_k$ for XCOMET from MQM labels, following the von Däniken et al. procedure. We then test whether $D_k$ against each candidate anchor correlates with it across systems. Three anchor families are considered. One is chrF, a surface overlap metric. One is a reasoning-model judge prompted in the ThinMQM style of Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost, which forces error-span annotation followed by fixed severity penalties. The third is a reference-free version of the same judge. Per the scale-input paradox reported there, small judges need a reference and judges of 32B parameters or more work best without one, so we test both modes.

The second stage builds the correction and evaluates it leave-one-system-out and leave-one-language-pair-out. The third stage checks transfer to WMT24 language pairs, where ThinMQM was evaluated, using only parameters fitted on WMT23. The main practical risk is sample size. With roughly 12–15 systems per pair, the predictor must stay at two to four coefficients, and we report bootstrap intervals over systems rather than point estimates.

Baselines

The naive reference point is uncorrected XCOMET mapped through the global function $g_m$. Its reported SysDep on WMT23 zh-en is 2.816 with segment correlation 0.65 (A Measure of the System Dependence of Automated Metrics). No reported en-de or he-en values for XCOMET, GEMBA-MQM or prismSrc appear in the file, so we recompute all of them with the published procedure. Beating it means a lower held-out SysDep, with bootstrap intervals that exclude the uncorrected value, while segment correlation does not drop.

The strongest label-free competitor is simple ensembling: average the human-scale scores of XCOMET and the same anchors with equal weights or weights fitted on training systems. It uses exactly the same inputs and compute as our method. If biases across families are independent, averaging alone reduces SysDep, so we must beat the fitted ensemble specifically, not the raw metric.

The strongest single-metric competitor is the ThinMQM-style judge used alone. The file reports correlation gains (+8.7 points at 7B, +3.9 at 32B on WMT24, reaching parity with xCOMET) but no SysDep figure; no reported result found on Lacuna for ThinMQM SysDep. We will measure it.

The label-based ceiling is a budget-matched recalibration. For each held-out system we draw $n$ random MQM-labeled segments, with $n\in\{25,50,100\}$, and fit an offset. This baseline tells us how many human labels our label-free correction is worth.

Two ablations isolate the key ingredient. In the first, we permute $D_k$ across systems. If the gain survives the permutation, it comes from global smoothing rather than system-specific signal. In the second, we replace each independent anchor with a metric sharing XCOMET's training data. This tests whether anchor independence matters.

Why not just…?

Why not just average several metrics? This is the real threat. If the biases of XCOMET, chrF and an LLM judge were independent and of similar size, averaging would capture most of the benefit with no fitting at all. Our method differs only if bias magnitudes vary by family and system, so that a weighted disagreement predicts the target's error better than a fixed mixture. The fitted-ensemble baseline settles this directly. If it matches us, we will report that ensembling is the practical answer and that the disagreement signal adds nothing.

Why not just annotate a few segments of each new system? When labels are affordable, this is correct, and it is our ceiling. The question is the exchange rate. AI-Assisted Human Evaluation of Machine Translation reports 71 seconds per error span for unassisted professional annotation. We cannot use its 31-second assisted protocol to cheapen this baseline, because that protocol cannot validate an automatic evaluator. We will report the label budget at which random annotation overtakes the label-free correction. If that budget is around 25 segments, the method's value is small and we will say so.

Why not just switch to a better judge, such as a calibrated reasoning model? Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost shows that calibrated judges improve correlation, but correlation is exactly the quantity that masked XCOMET's 2.816 SysDep. A better judge may simply have different system preferences, for instance the reported leniency bias. Measuring the judge's own SysDep, and correcting the judge using XCOMET as the anchor in the reverse direction, tests whether any single instrument escapes the problem.

Proposed Experiments and Decisive Checks

The first decisive check is the stage-one correlation between $D_k$ and true $\mathrm{ED}_k$ across the roughly 40 WMT23 systems. If its bootstrap interval includes zero for every anchor, the project stops there, and the negative result goes into the meta-evaluation literature. The second is held-out SysDep on zh-en against 2.816, compared with the fitted ensemble, together with whether Lan-BridgeMT's human rank is restored. The third is a sweep over the shrinkage factor $\lambda$. We predict an interior optimum: full correction should overshoot on systems whose anchors are themselves biased, mirroring the 10 mM versus 15 mM window. The fourth is transfer to WMT24 without refitting. The fifth is the permutation and shared-training-data ablations.

As an exploratory extension, LTBv1-eval (Last Translation Benchmark) could serve as a third, rule-based anchor for systems that can be queried. Its 911 human-authored verification rules are fully independent of continuous metrics, which are known to give scores above 80% to outputs that fail them. Because WMT23 submissions cannot be re-queried, this extension would apply only to open models we can run ourselves.