Quality estimation means predicting how good a machine translation is without a human reference translation to compare against. One family of methods, called glass-box quality estimation, looks inside the translation model and reads its own uncertainty: how spread out its word probabilities are, or how much its output changes when random noise (dropout) is switched on at inference time. These signals need no human labels and work surprisingly well. The other family trains a regression model on human quality scores, usually on top of a large pretrained multilingual encoder. It wins when labels are plentiful. For the language pairs that most need quality estimation, labels are scarce, and a large model fit to a few hundred scores tends to memorize them.
This note proposes treating the label-free glass-box score as the default prediction and letting the supervised model learn only a penalized correction on top of it. With very few labels the estimator should behave like a calibrated version of the unsupervised score. As labels accumulate it should approach the fully supervised system. The deliverable is less the estimator, which is deliberately simple, than a careful answer to a question the literature has not asked. At what label budget, for which language pairs, does supervised fusion of glass-box and text features stop overfitting and start helping? And does shrinking toward the glass-box score, rather than toward a constant, change that answer?
Core Claim
The existing results imply a crossover that nobody has measured. Unsupervised Quality Estimation for Neural Machine Translation extracts label-free signals from the translation model. Its strongest signal is D-TP, the average sentence probability over several dropout-perturbed forward passes. On Estonian–English this signal reaches Pearson r = 0.642 against human Direct Assessment scores (0–100 judgments), slightly above a supervised BERT-BiRNN at 0.635. The same signal struggles on high-resource pairs such as English–German, where translations are uniformly good. Beyond Glass-Box Features takes the opposite route. It concatenates many uncertainty features with XLM-R representations into a supervised regression head and trains on full labeled data, raising English–German from 0.3772 (TransQuest) to 0.5086 and Sinhala–English from 0.6207 to 0.6677. It never asks how that high-dimensional head behaves when labels are few. Evidence from outside translation suggests the risk is real. In Tackling the ADReSS Challenge, a multi-feature model on 108 training examples scored 92.7% cross-validation accuracy but 77.8% on the held-out test set, and linear models beat nonlinear ensembles.
Our claim has two parts. First, there is a label budget below which the fused supervised model is worse than the calibrated unsupervised score, and this budget differs sharply by language pair. Second, a residual estimator anchored to the glass-box score dominates both endpoints across the whole budget range. The key ingredient is the anchor. Ordinary regularization shrinks a small-data model toward predicting the mean, which is the wrong default when a label-free predictor already achieves r ≈ 0.6.
Research Program
Let $g(x)$ be a label-free glass-box score for translation $x$, such as D-TP or single-pass sequence log-probability. Let $\phi(x)$ be a feature vector concatenating frozen XLM-R sentence embeddings with the glass-box uncertainty features. Let $y$ be the human score. The estimator is
$$\hat y(x) = a + b\,g(x) + \theta^\top \phi(x),$$
fit by minimizing $\sum_i (y_i - \hat y(x_i))^2 + \lambda \|\theta\|^2$. The scalars $a$ and $b$ are unpenalized, and the penalty strength $\lambda$ is chosen by nested cross-validation inside the labeled sample only.
As $\lambda \to \infty$ the model reduces to a two-parameter calibration of the unsupervised score onto the human scale. That calibration is exactly what the unsupervised work lacks: it produces rankings, not 0–100 predictions. As $\lambda \to 0$ the model becomes a linear version of the fused supervised system. A second variant replaces the linear residual with full XLM-R fine-tuning, adding a penalty on the squared deviation of its predictions from $a + b\,g(x)$. This tests whether the anchoring effect survives in a high-capacity model.
The calibration idea is borrowed from Self-Evaluation of Large Language Model based on Glass-box Features, which reports that calibrating probability-based glass-box scores improves their ranking correlation for language-model self-evaluation. That paper calibrates against reference answers and does not study translation or sample size. We use the idea as one component, not as the contribution.
The analysis has three parts. The first is label-budget curves per language pair, at budgets from a few dozen sentences up to the full training split, each repeated over many random subsamples. The second is an explicit measurement of the gap between cross-validation and test performance, the failure mode documented in ADReSS, at every budget. The third is a comparison of anchors. We contrast the dropout-based D-TP, which is expensive at inference time, with cheap single-pass signals such as entropy and log-probability. If the cheap anchors work nearly as well, the approach also addresses the latency problem raised in both glass-box papers.
Baselines
The naive reference point is the calibrated unsupervised score alone. This is D-TP from Unsupervised Quality Estimation for Neural Machine Translation, with $a$ and $b$ fit on the same labeled subsample. Calibration leaves Pearson correlation unchanged, so on correlation this is literally the zero-label method, while also yielding absolute scores for error metrics. Its reported full-data figure is Pearson 0.642 on Estonian–English. Any supervised method that fails to beat it at a given budget is overfitting there.
The strongest existing method is the uncertainty-enhanced fused regressor of Beyond Glass-Box Features. It feeds XLM-R representations plus softmax, dropout, training-data and perturbation features into a feed-forward head. Its reported full-data results are Pearson 0.5086 on English–German and 0.6677 on Sinhala–English. TransQuest, the black-box system it beat (0.3772 and 0.6207), is a secondary baseline. Lacuna has no reported result for either system at reduced label budgets, so we rerun both. To keep the comparison fair, each gets the same subsample, the same glass-box features, and the same nested tuning budget over learning rate, epochs and head size as our estimator. Early stopping uses inner folds only.
We would count our method as beating these baselines if it shows significantly higher test Pearson across subsampling seeds at budgets up to a few hundred labels, with no significant loss at full data. Matching 0.5086 on English–German at full data is the bar for "no loss."
The black-box small-data competitor is MultiQE from Better Quality Estimation for Low Resource Corpus Mining. It fights the tendency of quality models to reward fluent but wrong translations by training on auxiliary semantic-similarity and entailment tasks, adding synthetic negative pairs, and using contrastive learning. It reports the top Pearson on 4 of 6 MLQE pairs. We run it on the same subsamples, keeping its auxiliary English data, since that data is part of the method.
Two ablations isolate the key ingredient. The first is shrink-to-zero: identical features and penalty, but with $g(x)$ treated as an ordinary penalized feature, which is plain ridge regression. The second is anchor-only: $\lambda \to \infty$, so the residual is removed entirely.
Why not just…?
Why not just use the unsupervised glass-box score and skip supervision? At tiny budgets this may be the right answer, and the experiments are designed to say where. The unsupervised paper itself shows weak performance on high-resource pairs like English–German, where supervised fusion more than doubles the gap over black-box baselines. A single zero-label score therefore leaves accuracy on the table there. The anchor-only ablation is exactly this alternative, and the budget curves show where it stops being sufficient.
Why not just run ridge regression on the glass-box features plus embeddings? This is the most serious threat, because our estimator is ridge regression with one unpenalized term. If shrink-to-zero matches the anchored version at every budget, the anchoring claim is false. The result would then reduce to an empirical finding about linear versus nonlinear heads, the same pattern ADReSS found, and we would report it as such. We expect a difference only at the smallest budgets. There, ridge with a large penalty pulls the coefficient on $g$ toward zero, discarding a predictor that already works. That is a falsifiable, specific prediction.
Why not just use MultiQE-style semantic transfer and stay black-box? MultiQE reduces the dependence on labels, but it borrows high-resource English supervision. Its gains also did not carry cleanly to the BUCC corpus-mining benchmark. Black-box transfer cannot see when the translation model itself was uncertain. Running it on the same subsamples, and adding $g(x)$ as an anchor on top of it, shows directly whether glass-box anchoring is complementary to semantic transfer or redundant with it.
Proposed Experiments
All experiments use MLQE-PE. This dataset released the internal states of the fairseq translation models alongside the human labels, which is what makes glass-box features available for every sentence. We use English–German and English–Japanese (high-resource or hard), Romanian–English and Estonian–English (mid-resource), and Sinhala–English and Nepali–English (low-resource). For reference, the dataset's own XLM-R baseline ranges from 0.818 Pearson on Romanian–English to 0.230 on English–Japanese.
The primary target is Direct Assessment. We repeat the experiments with edit-rate labels (HTER, the fraction of words a human changed when post-editing), because the dataset paper documents that the two targets diverge. Training subsamples are drawn at budgets of 50, 100, 200, 500 and 1000 sentences, plus the full split, with 20 seeds each. Metrics are test Pearson, mean absolute error on the 0–100 scale (meaningful only because of calibration), and the cross-validation-minus-test gap.
The decisive checks are four. First, the crossover budget at which the fused regressor overtakes calibrated D-TP should be measurably larger for low-resource pairs than for English–German. Second, the anchored estimator should be at or above the better of the two endpoints at every budget. Third, anchored should beat shrink-to-zero at small budgets. If it does not, we drop the anchoring claim. Fourth, a single-pass anchor should recover most of the D-TP anchor's benefit. That would make the method usable without multiple dropout passes at inference.
References
- Unsupervised Quality Estimation for Neural Machine Translation
- Beyond Glass-Box Features: Uncertainty Quantification Enhanced Quality Estimation for Neural Machine Translation
- Tackling the ADReSS Challenge: A Multimodal Approach to the Automated Recognition of Alzheimer’s Dementia
- Self-Evaluation of Large Language Model based on Glass-box Features
- Better Quality Estimation for Low Resource Corpus Mining
- MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset