Teaching Reference-Free Translation Scorers When a Mismatched Number Matters

Automatic translation quality estimation predicts how good a machine translation is from only the source sentence and the output, without a human reference translation. Current estimators are large multilingual neural networks that compare whole-sentence meaning. They agree well with human judges on average, yet they barely react when a translation swaps "29% yes, 31% no" into "31% yes, 29% no", changes a unit, or garbles a date. The reason is simple: the sentence still means almost the same thing as a whole. A few cheap checks on the visible text can spot these errors directly, for example whether the numbers, entities and units in the source reappear, correctly placed, in the output. The difficulty is that such checks also fire on harmless differences such as "15" versus "fifteen", "1,000" versus "1.000", or a correct unit conversion. Penalizing every mismatch therefore trades one failure for another.

We propose to sit a small gradient-boosted tree ensemble (many shallow decision trees, each correcting the errors of the ones before it) on top of a frozen neural quality estimator that we treat as a black box. The ensemble sees the neural score together with explicit source–target consistency features. It is trained to learn when a surface mismatch should lower the score and when it should be ignored. Natural human-scored data teaches it which mismatches are harmless. Synthetic meaning-destroying edits teach it which are fatal. The claim is about interactions: a mismatch's importance depends on its type, its position, and what the neural score already says. A model that splits on hard thresholds can represent this kind of conditional rule cheaply, but a fixed rule cannot. Every competing head receives an identical grid-search tuning budget, and a sweep over tree depth directly tests whether the interactions exist.

Core Claim

A tree-based correction layer over a black-box neural quality estimator can repair its blindness to numbers, dates, units and entities while keeping its correlation with human judgments on natural data. It can also do this with fewer false alarms on meaning-preserving formatting variants than (a) a tuned "penalize any mismatch" rule, (b) a linear model on the same features, or (c) fine-tuning the neural estimator on the same synthetic edits. The central falsifiable prediction is that trees of depth two or more beat depth-one trees, which can only add independent per-feature penalties, on the combination of three things: error detection, benign-variant tolerance, and natural correlation. If depth one is as good, the interaction story is wrong, and a calibrated additive rule is all practitioners need. That would be a useful negative result, and we would report it as such.

Where This Sits in Prior Work

The motivating evidence comes from two results that pull against each other. Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task introduced COMET-QE, a dual-encoder regressor over multilingual transformer embeddings. It showed that reference-free scoring can match reference-based scoring, reaching Pearson 0.720 against MQM (expert error-annotation) scores on WMT21 Chinese–English. Exploring Robustness of Machine Translation Metrics then built a 721-case adversarial Chinese–English challenge set. On it, 20 of 28 metrics stayed below a 0.4 Kendall-style correlation, and almost all failed to penalize number and unit swaps. That paper is purely diagnostic and proposes no repair. Our work takes its failure categories as the target.

Two older systems supply the method. FBK-UPV-UEdin participation in the WMT14 Quality Estimation shared-task regressed post-editing effort on a large, collinear pool of hand-built features with Extremely Randomized Trees. It showed that tree ensembles absorb heterogeneous quality features well, but it used pre-neural statistical features, did not tune the trees systematically, and never probed micro-errors. Black Box Features for the WMT 2012 Quality Estimation Shared Task showed that cheap string features such as named-entity tags, punctuation, and edit distances carry individual signal (word edit distance reached PCC 0.32). Combined naively, however, they were "nearly indistinguishable" from the baseline. That result constrains our design: surface features alone are not the answer, and the question is how they combine with a strong neural score.

Target-Centric Features for Translation Quality Estimation constrains our evaluation protocol. Its quality estimator failed at competition time with default settings and became competitive (MAE 0.55 on English–German) only after a post-hoc grid search. Untuned comparisons in this area are therefore unreliable, so we tune every head under one budget. Rethinking Round-Trip Translation for Machine Translation Evaluation shows that translating the output back into the source language gives a reference-free fidelity signal. We use it only as one more input feature.

The closest framing of our idea is the hybrid-feature tree suggested by combining FBK-UPV-UEdin with Hokamp et al. Our proposal differs from it in three ways. First, the neural estimator is frozen and treated as one input, not replaced. Second, the training signal mixes natural human scores with pairwise constraints from synthetic edits. Third, success is defined jointly on adversarial detection, benign-variant tolerance and natural correlation, not on average correlation alone.

Research Program

The base scorer is a released COMET-QE-style model. It is queried only for its output score, which keeps the setup black-box with respect to both the translation system and the scorer. We will also run a distilled variant of the kind described in the Unbabel-IST paper to check that conclusions survive a weaker base.

On top of the base score we extract consistency features from the source and output strings. The number features are built in three steps. Numbers are normalized: digits, number words, and thousand and decimal separators are mapped to one form, and dates are canonicalized. We then record set agreement between source and output numbers. Finally, we record an order- and anchor-aware agreement: each number is paired with its neighboring content words through a word aligner, of the kind whose adequacy value FBK-UPV-UEdin demonstrated. This is what catches the 29%/31% swap, where the two sets are identical but the pairing is wrong. The remaining features follow the same approach. Unit tokens are matched against a small conversion table, so a correct kilometre-to-mile conversion counts as agreement. Named entities are matched with transliteration-tolerant string similarity. Negation markers are counted on each side. The input set is completed by length ratio and a round-trip divergence score.

Training uses two data sources. Natural segments with human MQM or Direct Assessment (rating-scale) scores from the WMT21/22 MLQE-PE data enter with a squared-error loss. This is where the model learns that many natural mismatches are harmless, because humans rated those translations well. Synthetic contrast pairs are made by applying meaning-destroying edits to good translations: swapping numbers, shifting dates, replacing units, corrupting entities, and flipping negation. These enter only as a hinge constraint that the edited version must score at least a margin $\delta$ below its original. Using a constraint avoids inventing an absolute label for synthetic data, which would otherwise build a fixed penalty into the target and make the rule baseline equal by construction. We implement this as gradient boosting with a custom loss:

$$\mathcal{L} = \frac{1}{|N|}\sum_{i \in N} (f(x_i) - y_i)^2 + \frac{\alpha}{|P|}\sum_{(g,b) \in P} \max\big(0,\ \delta - f(x_g) + f(x_b)\big)$$

Here $f$ is the ensemble's corrected score, $N$ the natural segments with human score $y_i$, and $P$ the pairs of good translation $g$ and edited translation $b$. The weight $\alpha$ and margin $\delta$ are hyperparameters.

Tuning is a coarse-to-fine grid with five folds grouped by source sentence, so that edited and original versions never straddle folds. The grid covers tree depth, number of trees, learning rate, minimum samples per leaf (which matters because fatal errors are rare in natural data), $\alpha$ and $\delta$. The selection objective is the natural-data correlation subject to a minimum pair accuracy on held-out synthetic pairs. This is fixed before any test set is touched.

The depth sweep is analyzed as a scientific result, not just a tuning step. Depth one means additive per-feature corrections; depth two and above admit interactions. For the selected model we will also report the most-used split paths. One example is "anchored-number disagreement AND base score above threshold", which would indicate that the tree corrects confident neural mistakes specifically. These paths are checked against the edit category.

Baselines

The naive reference point is the frozen neural estimator alone, with no correction. On WMT21 Chinese–English MQM it is reported at Pearson 0.720 (Unbabel-IST 2021). Matching that figure on our evaluation split is the "do no harm" bar: a correction that drops natural correlation by more than run-to-run noise, estimated by bootstrap, fails regardless of its adversarial gains. On the robustness challenge set, the file reports only aggregate failure rates. There is no reported result found on Lacuna for COMET-QE's individual score on that set, so we will measure it ourselves.

The strongest simple competitor is a tuned override rule. It subtracts a fixed penalty from the neural score for each detected mismatch type, using exactly our extracted features, with per-type penalties chosen by the same grid search and folds. This is what a practitioner would ship, and it directly tests whether learning adds anything. Beating it means equal or better detection on the challenge set together with a lower false-alarm rate on benign variants. Detection alone is not enough, because a large enough penalty always catches swaps.

A linear stacking model, ridge regression on the same features with the same loss and grid budget, tests whether thresholds and interactions are needed at all. It takes the role that Partial Least Squares played in Estimating the Sentence-Level Quality of Machine Translation Systems, where linear regression over black-box features clearly outperformed BLEU-2 (0.602 vs. 0.342). The depth-one boosted model is the ablation that isolates the key ingredient: it keeps the tree learner but removes interactions. To connect to the literature's untuned practice, we will also run Extra-Trees with library defaults on the same features, in the style of FBK-UPV-UEdin.

The strongest neural competitor is the base estimator fine-tuned on the same natural data plus the same synthetic pairs and the same loss. Its grid covers learning rate, epochs and augmentation ratio, with the same number of configurations as the tree heads. Its compute is far higher, and we will report it rather than hide it. Beating it means matching its detection and natural correlation while generalizing better to held-out edit categories, at a small fraction of the training and inference cost.

For catastrophic error detection, the reference number is MCC 0.546 for English–German by the best WMT21 Task 3 system (Findings of the WMT 2021 Shared Task on Quality Estimation). We will compare against it only on the number and named-entity subcategories that our features address, and we will state that restriction openly.

Why not just…?

Why not just add a hand-written penalty whenever numbers or entities disagree? This is the real threat, and the proposal is built so that it can win. Its known weakness is false alarms: Zhu et al. document metrics confusing "15" with "fifteen", and natural data contains correct conversions and reformattings. Its strength is simplicity. The tuned rule baseline, combined with a dedicated benign-variant test set, settles the question. If the rule matches the tree on both detection and false alarms, we conclude that the consistency features, not the learner, carry the gain, and we recommend the rule.

Why not just fine-tune the neural estimator on synthetic number swaps? It may work on the edit types it sees. The risk is that a sentence-level encoder learns the generator's artifacts rather than a general check for whether numbers agree. Untargeted fine-tuning has also produced the very models that Zhu et al. found brittle. The leave-one-category-out experiment below tests this directly. If fine-tuning generalizes as well as the tree, the remaining argument for the tree is cost and inspectable decision paths. We will say so plainly rather than overstate it.

Why not just use a linear combination of the neural score and the features? If mismatches matter independently of context, a linear model is enough and easier to trust. Specia et al. show that linear models over black-box features are already strong. The ridge and depth-one baselines answer this question. The interaction claim survives only if depth two or more beats both on the joint criterion.

Why not just prompt a large language model to check the translation? Lyu et al. (A Paradigm Shift) argue that prompted models catch nuances that embedding comparisons miss, and they may well catch swapped numbers. They are, however, orders of magnitude slower and costlier per segment, and the file reports no result for them on the challenge set. We will run one prompted model on the 721 challenge cases and the benign set as an upper reference, but not as a production competitor.

Proposed Experiments and Decisive Checks

The adversarial test is the full 721-case Chinese–English challenge set from Zhu et al., used only at test time and scored with its pairwise ranking accuracy, broken down by category. We will construct our synthetic training edits independently of that set.

The benign-variant set will be built by applying meaning-preserving transformations to the same source segments: number words versus digits, separator conventions, and correct unit conversions. We score it by the rate at which the model rates the variant clearly below the original. Natural correlation is measured as Pearson against MQM on WMT21 Chinese–English and against Direct Assessment on MLQE-PE pairs, with bootstrap confidence intervals. Catastrophic detection uses MCC on the WMT21 Task 3 number and named-entity subsets and on SMAUG from Findings of the WMT 2022 Shared Task on Quality Estimation.

The first decisive check is the depth sweep under equal tuning budgets. Interactions are supported only if the best depth-two-or-more model dominates depth one and ridge on challenge accuracy and benign false alarms, without losing natural correlation.

The second is leave-one-category-out training. We train on edits from four of the five categories (numbers, dates, units, entities, negation) and test on the fifth, for the tree, the rule and the fine-tuned network. This shows whether the correction learns "trust the neural score unless an anchored check disagrees" or merely memorizes edit types. A tree that does not transfer to unseen categories better than fine-tuning loses its main advantage.

The third is a base-model swap from the full estimator to its distilled variant. If the correction mostly recovers what the weaker base misses, its gains should grow there. If the gains vanish on the stronger base, the micro-error problem is smaller than Zhu et al. suggest for modern estimators.