Automatic post-editing (APE) trains a model to fix the errors a machine translation system makes, so that human translators do less repetitive correction. As translation systems have improved, APE has developed a characteristic failure: it rewrites sentences that were already right. In the WMT 2023 English–Marathi task, an LLM-prompting system edited over 90% of sentences, improved only about 20% of them, and pushed corpus Translation Edit Rate (TER, the fraction of words a human would need to change, lower is better) from 26.60 to 54.59 (Findings of the WMT 2023 Shared Task on Automatic Post-Editing). Current defenses all ask the same question: is this piece of the translation wrong? Some ask it for the whole sentence, some for each word, and some through a coarse effort label. None of them asks whether a particular proposed fix is actually better than what it replaces.
We propose to ask that second question directly. A span-level corrector proposes candidate edits, each a local replacement of a short stretch of the draft. A separate edit-level quality estimator then scores each (source, draft span, proposed replacement) triple for the probability that applying it lowers TER against a human post-edit. Edits are accepted one at a time, in score order, across the whole test corpus. Moving the acceptance cutoff traces the entire trade-off between correcting more errors and damaging fewer good sentences. That trade-off comes from one decoding pass, with no retraining and no constrained beam search, and it is evaluated as a curve across domains rather than as a single point on one shared-task set.
Core Claim
Over-correction happens because the model has no signal for whether its own edits help. Quality estimation (QE, predicting translation quality without a reference) of the draft is the wrong target for this. A draft token can be genuinely wrong while the proposed replacement is also wrong, or wronger. Our claim has two parts. First, a scorer trained on edit-level labels ranks proposed edits better than the word-level BAD probabilities that current QE-guided systems use. Second, on the resulting curve of TER against deterioration rate (the fraction of sentences made worse than the raw MT), the edit scorer dominates threshold sweeps of existing gating methods. The advantage should be largest at the conservative end, where production systems must operate, and it should survive a move to a different domain.
The labels cost nothing extra. For every training triplet we apply each candidate edit to the draft in isolation and compute whether TER against the human post-edit falls. The 7K–18K human triplets per language pair used in prior WMT work (Giving the Old a Fresh Spin) therefore yield many more edit-level examples than sentence-level ones, which matters given the known data-hunger of NMT post-editing (Can Automatic Post-Editing Improve NMT?).
How This Relates to Prior Work
Three lines of work motivate the gap by attacking over-correction without edit-level feedback.
- Effort-Aware Neural Automatic Post-Editing conditions a dual-encoder model on a three-way effort token. The oracle token helps, but predicted tokens fall below the baseline, and no system beat raw MT on WMT19 English–Russian. That result suggests the decision must be made at a finer grain than a sentence-level category.
- Quality Estimation-Assisted Automatic Post-Editing trains APE jointly with sentence and word QE, using Nash-MTL (a bargaining method for reconciling conflicting training gradients). It reports gains of 1.09 TER on English–Marathi and 0.46 TER on English–German over baseline APE. It also finds word-level tags to be the decisive ingredient, which supports our move toward finer granularity.
- Giving the Old a Fresh Spin locks QE-"OK" words as hard constraints in Grid Beam Search. This buys 0.65, 1.86 and 1.44 TER on English–German, English–Hindi and English–Marathi, at the cost of heavy decoding latency and permanent lock-in of any mislabeled token.
All three score the draft. We score the change.
The mechanism reuses Detector-Corrector, which generates text only inside detected error spans and reaches TER 27.7 on WMT20 English–German, against 28.4 for sequence-to-sequence APE and 31.3 for raw MT. Its oracle experiment shows over 20 TER points separating predicted from true error tags. We read that gap as evidence that the bottleneck is deciding which edits to keep, and we insert our scorer exactly there, after correction rather than before it.
The multi-objective literature constrains the design more than it supplies it.
- Preference-Based Policy Iteration and Navigating Conflicting Objectives with Pareto Optimization warn that a single weighted sum hides non-convex parts of a trade-off. We therefore report whole curves and check greedy selection against exact enumeration.
- Multi-Objective Optimization Through Pareto Minimal Correction Subsets shows that Pareto-optimal solutions of discrete selection problems can be found as minimal subsets of "corrections". This matches our setting almost literally: choose the smallest set of edits that achieves a given quality. It motivates the exact-enumeration check.
We deliberately do not train a preference-conditioned universal model in the style of PD-MORL. Once edits are scored independently, the curve is obtained by sorting, and a learned preference interpolator would add drift without adding reach.
Finally, WMT 2020 shows that APE gains depend heavily on domain: the IT data had a repetition rate of 18.25, against 0.653 for the Wikipedia data (Findings of the WMT 2020 Shared Task). The image-processing literature in our file repeatedly shows methods tuned on tiny curated sets failing on new textures and domains. Together these make single-domain, single-point evaluation the main reason to distrust any over-correction result, and our evaluation is built around that concern.
Research Program
We will build three components.
The first is a span corrector. It uses a Detector-Corrector-style shared XLM-RoBERTa encoder with a span decoder, run at a deliberately high recall setting so that it proposes many candidate edits per sentence, including edits inside spans the detector is unsure about. When two edits interact, for example an agreement fix touching two spans, the corrector emits them together as one joint candidate. Overlapping candidates are treated as mutually exclusive.
The second is the edit scorer. It is a cross-encoder over the source, the draft with the span marked, and the proposed replacement. It is trained on the automatic "does this edit alone lower TER" labels from human triplets, and optionally pre-trained on synthetic triplets. Because synthetic error distributions differ from human ones, we report results with and without the synthetic stage.
The third is corpus-level selection. Let $E$ be all candidate edits over a test set $D$, and let $p(e)$ be the scorer's probability that edit $e$ helps. For a harm budget $\epsilon$ we select
$$S_\epsilon = \arg\max_{S \subseteq E \text{ non-overlapping}} \sum_{e \in S} p(e) \quad \text{subject to} \quad \sum_{e \in S} (1 - p(e)) \le \epsilon\,|D| .$$
Greedy acceptance in decreasing $p(e)$ approximately solves this, and sweeping $\epsilon$ traces the curve. Because the budget is shared across the corpus, the near-perfect peak of the U-shaped difficulty distribution naturally receives almost no edits. The sentences in the 10–40 TER middle band receive most of them.
The efficiency claim follows from the design. The corrector decodes only short spans once, and every operating point is then a re-sort of cached scores. The whole curve therefore costs roughly one span-decoding pass plus one scorer pass per candidate, whereas Grid Beam Search must re-decode per constraint set. No reported latency figure for Grid Beam Search in APE was found on Lacuna, so we will measure wall-clock time ourselves on shared hardware.
Baselines
The naive reference point is raw MT, the "do nothing" system. On WMT23 English–Marathi it scores TER 26.60, BLEU 70.66 and chrF 79.78 (WMT 2023 Findings), and by definition it has zero deterioration. Any curve point must beat it on TER at a stated deterioration rate. The best officially reported English–Marathi figure is HW_TSC's late submission at TER 25.74, which gives the scale of achievable improvement.
The strongest existing method is QE-assisted constrained decoding (Giving the Old a Fresh Spin), with reported TER gains of 1.44 on English–Marathi, 1.86 on English–Hindi and 0.65 on English–German. We will turn it into a curve by sweeping the word-QE probability threshold used to lock tokens. It will use the same XLM-R QE model, the same human and synthetic training data, and the same underlying APE model where applicable. We will also report its latency. Beating it means a curve that lies below it in TER at every deterioration level up to 10%, confirmed by paired bootstrap resampling at the matched deterioration rate, at lower decoding cost.
For WMT20 English–German we compare against Detector-Corrector's reported TER 27.7 (Detector-Corrector) and against our re-implementation swept over its detection threshold. We also compare against Nash-MTL joint training (QE-Assisted APE) as a strong single-point system, extended to a curve by gating with its sentence-QE head.
The key ablation keeps our corrector and our selection procedure fixed and replaces $p(e)$ with the word-level BAD probability of the span being replaced. This isolates the one ingredient we claim is new: scoring the replacement rather than the draft. A second ablation replaces corpus-level budgets with per-sentence thresholds, to test whether sharing the budget across sentences matters for U-shaped data.
All systems receive the same human triplets, synthetic data, backbone size and hyperparameter search budget, which we fix at 20 development-set trials per system.
Why not just…?
Why not just gate standard APE with a sentence-level QE threshold? Deoghare et al. already found that QE-as-activator and QE-as-selector pipelines underperform joint training on both English–Marathi and English–German (QE-Assisted APE). A sentence gate also cannot save the good half of a bad sentence. We still include it as a swept curve, since it is cheap and its conservative end may be competitive on near-perfect-heavy data like WMT23. If it matches us there, we will say so.
Why not just sweep the word-QE threshold in the existing constrained decoder? This is the real threat. It already yields a trade-off curve, and word QE is a mature model: MLQE-PE baselines reach an average F1-BAD of 0.579 (MLQE-PE). Our argument rests on the fact that "this token is wrong" does not imply "this fix is right", and that argument is only as good as the ablation that swaps word-BAD probability for edit scores inside an otherwise identical system. If that ablation shows no gap, then the contribution reduces to span-restricted decoding for speed. We would report that rather than dress it up.
Why not just prompt an LLM to post-edit? The KAIST GPT-3.5 submission degraded English–Marathi from 26.60 to 54.59 TER (WMT 2023 Findings). But LLM rewrites could still be a good source of candidate edits if something filters them. We therefore run an open instruction-tuned LLM as an alternative candidate generator, split its rewrites into span edits by alignment, and pass them through our scorer. If filtered LLM edits beat our corrector's edits, that is a useful finding for the proposal rather than a refutation of it.
Why not just use effort tokens or a weighted preservation loss? Predicted effort tokens underperform even the token-free baseline (Effort-Aware APE). A fixed loss weight commits to one operating point at training time and can miss non-convex regions of the curve (Preference-Based Policy Iteration). We include a copy-loss-weighted APE model trained at three weights as extra points on the comparison plot.
Proposed Experiments
In-domain curves. We train per pair on WMT20 English–German and English–Chinese (Wikipedia), WMT21/22 English–Marathi and English–Hindi, and WMT23 English–Marathi. Each system is plotted as TER and chrF against deterioration rate. For a single summary number we use the area between the system's curve and raw-MT TER over deterioration rates from 0 to 10%.
Domain shift. Evaluation scope is the axis most prior APE papers skip, so we test it directly in two ways. First, English–German models trained on WMT20 Wikipedia triplets are evaluated on the SubEdits subtitle post-edits, which contain over 161K English–German triplets (Can Automatic Post-Editing Improve NMT?). We also run the reverse direction, training on a 7K SubEdits subsample to match WMT scale. Second, we use the MLQE-PE post-edits to test transfer to its low-resource pairs, for example Nepali–English and Sinhala–English. The question is whether the ranking of methods, not just their absolute TER, is preserved under shift. The specific prediction is that edit-level scores, which condition on the replacement, degrade less than draft-level QE, whose calibration is known to vary widely across pairs: MLQE-PE sentence-level Pearson correlations range from 0.818 to 0.230.
Front fidelity. On development sentences with at most 12 candidate edits, we enumerate all subsets exactly and measure how far greedy selection falls from the true curve. This tells us whether the non-convexity warnings from the Pareto literature bite in practice.
Cost. We measure wall-clock time per 1,000 sentences on identical GPUs for Grid Beam Search, full beam APE, and our pipeline, where our cost covers the complete curve rather than one point.
Decisive checks. Three results decide the claim:
- At deterioration of 2% or less on WMT23 English–Marathi, the edit scorer achieves TER strictly below 26.60 while the swept word-QE constrained decoder does not beat it at the same deterioration.
- The edit-score versus word-BAD ablation shows a significant gap on at least three of the five in-domain pairs.
- That gap persists on the SubEdits and MLQE-PE transfer evaluations.
If check 2 fails, the conceptual claim is wrong, and the work stands only as an efficiency result.
References
- Findings of the WMT 2023 Shared Task on Automatic Post-Editing
- Giving the Old a Fresh Spin: Quality Estimation-Assisted Constrained Decoding for Automatic Post-Editing
- Can Automatic Post-Editing Improve NMT?
- Effort-Aware Neural Automatic Post-Editing
- Quality Estimation-Assisted Automatic Post-Editing
- Detector-Corrector: Edit-Based Automatic Post Editing for Human Post Editing
- Preference-Based Policy Iteration: Leveraging Preference Learning for Reinforcement Learning
- Optimal Scalarizations for Sublinear Hypervolume Regret
- Multi-Objective Optimization Through Pareto Minimal Correction Subsets
- PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm
- Findings of the WMT 2020 Shared Task on Automatic Post-Editing
- MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset