Judging machine translation still depends on professional translators marking errors by hand, and that work is slow and expensive. A recent shortcut has a language model pre-mark likely errors so the human only corrects its suggestions. This roughly halves annotation time. It also creates a risk: the human may start agreeing with the machine instead of judging the translation. If the machine shares blind spots with the translation systems under test, the human scores inherit those blind spots. We propose to treat the amount of AI pre-annotation as a dose. We would sweep it on a deliberate grid and plant known true and false errors at every level. That lets us measure where AI help stops correcting human noise and starts biasing human judgment.
The design comes from materials science. Engineers who heat-treat freshly deposited films, a step called post-deposition annealing, do not assume more heat is better. They sweep temperature across a range to find the operating window. One study found that 550 °C gave the desired vanadium dioxide phase, while 650 °C turned the film into a useless oxide (vanadium oxide annealing study). They also inject a calibrated reference signal so they can separate the effect they care about from other effects layered on top of it (lithium niobate waveguide study). We borrow both practices, the dose sweep and the calibrated probe, and apply them to human evaluation. We borrow nothing from the physics itself. The file contains no prior work joining the two fields. The connection is purely about experimental design, and we say so plainly.
Core Claim
AI-assisted error annotation has an interior optimum. Showing the human every machine-suggested error span is not the most cost-effective setting once validity is measured properly. Validity here means agreement with an independent expert reference, ranking systems correctly, and not favoring systems related to the assisting model.
The specific new claims are two. First, the human's tendency to defer to the machine can be measured directly at each dose. We do this by planting both missed true errors and fabricated false errors, then recording how often humans fix each kind. Second, the bias introduced by assistance is system-dependent. Translation systems that share lineage with the assisting model should be over-rated as the dose rises. This can be quantified with the system-dependence measure that was previously defined only for automatic metrics.
Prior Work and Its Role
The method we build on is the protocol from AI-Assisted Human Evaluation of Machine Translation. In that work, a GPT-4-based quality estimator (a model that scores a translation without a reference) pre-fills error spans. Professional translators then edit those spans and give a 0–100 score. On WMT23 English→German, this cut time per segment from 71 s to 31 s. It raised inter-annotator Spearman correlation from 0.376 to 0.533 and raised found errors from 0.5 to 1.6 per segment. The authors injected missed errors as attention checks and concluded that annotators did not blindly defer.
The authors also flag the open risk we study: the assisting model and the evaluated systems may share blind spots. Their study compares one dose, full pre-fill, against zero. It probes deference only in one direction. It does not vary how closely the assisting model is related to the systems being judged.
Two results motivate our system-level focus. A Measure of the System Dependence of Automated Metrics fits a separate isotonic regression from metric score to human score for each system. It defines SysDep as the worst-case expected deviation across systems. XCOMET reaches 0.65 segment correlation on zh-en yet has SysDep 2.816, and it underrates Lan-BridgeMT. We reuse this measure to audit human protocols, which no one has done.
Results of WMT23 Metrics Shared Task shows that the "ground truth" can itself be flawed. The zh-en human reference scored MQM error 4.83 against 2.10 for the best system. This is why our probes are planted, known errors rather than reference comparisons.
Two works constrain the design. WMT21 Metrics results report that neural metrics miss reversed negation and degrade from news to TED talks. That tells us which planted errors the assisting model is likely to miss. Don't Sweat the Small Stuff introduces Pairwise Difference Pearson, which compares two translations of the same source. This is a useful segment-level criterion, because assistance may shift scores uniformly while still preserving within-source orderings.
Research Program
The assisting model is a GEMBA-style quality estimator. For each candidate error span, we measure confidence as the fraction of $k$ sampled runs that propose that span. A dose $\tau$ means the annotator sees only spans with confidence at least $\tau$. We use five levels: $\tau=1$ shows nothing (plain error-span annotation), $\tau$ near 0 shows everything ever proposed, and three levels sit in between. We also vary the assisting model's lineage, crossed with dose. One version comes from the same model family as some of the evaluated systems. The other comes from an unrelated family.
Every batch contains two kinds of calibrated probes. Planted true errors are edits such as negation flips, number changes and dropped clauses, inserted into otherwise clean outputs and hidden from the pre-fill. Planted false spans are fabricated suggestions on correct text. The rate at which humans add the missed errors and the rate at which they delete the fake ones give two separate deference estimates.
Each protocol cell then has four outcomes. Cost is seconds per segment. Detection is recall on planted errors and rejection rate on fake spans. Agreement is measured against the existing WMT23 MQM annotations and between annotators. System-level validity is Kendall tau of the system ranking against MQM, plus SysDep computed with human protocol scores in place of a metric.
From these we trace a cost–validity curve and locate its knee. The analogue of the materials "process window" is the dose range where cost falls while system-level validity stays within MQM's own resampling noise.
Baselines
The do-nothing reference is plain human error-span annotation with no pre-fill, which is $\tau=1$ in our grid. The prior paper reports 71 s per segment and inter-annotator Spearman 0.376 on WMT23 en-de (AI-Assisted Human Evaluation). We rerun it with the same translators, segments and interface, so the comparison is not against a historical number.
The strongest existing human protocol is full pre-fill ESA^AI, the near-zero-$\tau$ arm. The same paper reports 31 s per segment, Spearman 0.533 and 1.6 errors per segment. To count as beating it, an intermediate dose must do two things. It must match its system-ranking accuracy against MQM within resampling noise while costing no more than about 10% extra time. Or it must cut its lineage-specific SysDep and its fake-span acceptance at equal cost. We also add the prior paper's zero-error pre-filtering, reported at up to 25% budget savings, as a cost-reduction baseline. We run it on the same segments.
The strongest fully automatic competitor is XCOMET. It reaches 0.825 average correlation with humans in WMT23 and has SysDep 2.816 on zh-en (System Dependence). We score the same segments with it and compare its ranking and SysDep against each human arm. We expect it to be cheapest and least valid.
The key ablation separates dose from lineage. We hold dose fixed and swap only the assisting model's family. If system-favoring bias tracks lineage and not dose, that is the confound to report. If it grows with dose only for the related family, it confirms the core claim.
Why not just…?
Why not just use full pre-fill, since it already halves cost and raises agreement? Higher agreement is exactly what priming would also produce, whether or not the shared answer is correct. The prior paper's attention checks tested only one direction of deference and one dose. This is the real threat to our hypothesis. If the full-pre-fill arm shows fake-span rejection and lineage SysDep indistinguishable from the zero-dose arm, full pre-fill wins. We would report that as the result.
Why not just replace humans with XCOMET or a similar metric? Its documented system dependence, 2.816 SysDep on zh-en, and its documented blindness to negation reversal (WMT21) are the very failures our human protocol is meant to catch. We include it in the comparison so the size of that gap is measured on the same data rather than assumed.
Why not just debias afterwards with a statistical rater model? Post-hoc rater models can correct per-annotator offsets. They cannot recover errors that no annotator marked because the pre-fill steered everyone away from them. Our planted-error recall tells us whether such missing information exists. If recall is flat across doses, post-hoc correction would suffice, and we would recommend it instead.
Why not just annotate fewer segments? Subsampling and zero-error filtering cut cost without touching how each judgment is made. They are complementary to choosing a dose, not a replacement. We report them both alone and combined with the chosen dose.
Proposed Experiments
The main study uses WMT23 en-de, where the ESA^AI numbers exist. It also uses zh-en, where SysDep and the Lan-BridgeMT inversion are documented. Both have expert MQM annotations to serve as an independent reference. Translators are randomized to dose and lineage cells in a counterbalanced design, so that order and fatigue effects can be estimated.
A smaller transfer study runs on IndicMT Eval (IndicMT Eval). There, quality estimators are weaker and experts scarcer, so the optimal dose should move. We test whether the knee measured on a high-resource pair predicts the knee on a lower-resource one.
The decisive check has three possible outcomes. The proposal succeeds if two things both hold: some intermediate dose sits on the cost–validity frontier strictly better than both endpoints, and the related-lineage arm shows SysDep rising with dose while the unrelated arm does not. If fake-span acceptance and SysDep are flat across all doses, the risk the prior paper flagged is empirically small, and full pre-fill is vindicated. If even low doses raise SysDep, AI pre-annotation should not be used for ranking systems from the same family. Any of the three outcomes would give evaluators guidance they lack today.
References
- Microstructurally optimizing the mid-infrared optical modulation properties of vanadium oxide thin films via magnetron sputtering and subsequent annealing
- Reduced material loss in thin-film lithium niobate waveguides
- AI-Assisted Human Evaluation of Machine Translation
- A Measure of the System Dependence of Automated Metrics
- Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent
- Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain
- Don’t Sweat the Small Stuff: Segment-Level Meta-Evaluation Based on Pairwise Difference Correlation
- IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for Indian Languages