A translation cascade runs a cheap model on every sentence and calls an expensive model only when a quality check says the cheap translation is probably bad. The best current version, Translate Smart, not Hard, matches a 70B model's quality while sending only 30% to 50% of sentences to it. Its escalation step, however, is all-or-nothing. When a sentence is deferred, the cheap translation is thrown away and the large model starts from scratch. Hybrid translation research has spent decades showing that a second engine is often more useful as a corrector of the first engine's output than as a replacement for it. That line runs from statistical post-editing of rule-based output (Statistical Phrase-Based Post-Editing) to GPT-4 editing neural translations (Leveraging GPT-4 for Automatic Translation Post-Editing). That literature also shows that correcting everything is harmful: editors damage translations that were already fine.
We propose a deferral gate with three choices instead of two. For each sentence it decides whether to keep the cheap draft, have the large model repair only the spans flagged as wrong, or have the large model retranslate from scratch. The gate does not trigger on low absolute quality. It predicts how much each choice would improve the final translation and weighs that gain against the choice's cost. In two sentences: current translation cascades ask "is the small model's output bad?", and we ask "which kind of large-model help would actually fix it, and is the fix worth the price?" This should save budget on sentences the large model cannot fix either, such as garbled or ambiguous sources. It should also recover quality on sentences where the cheap draft is mostly right and a full retranslation would introduce new errors.
Core Claim
Our central claim is that in translation cascades, deferral should depend on the predicted gain from each escalation action, not on the small model's quality alone. Adding a cheap-to-decide "repair flagged spans" action between "keep" and "retranslate" moves the cost–quality curve beyond what binary deferral can reach. Two facts in the file motivate this.
The first is that large translation models do not dominate small ones. Tower-70B beat Tower-7B on only 43% of WMT24 sentences (Translate Smart, not Hard). So retranslation regularly throws away a draft that was as good as or better than what replaces it.
The second is theoretical. For classification, When Does Confidence-Based Cascade Deferral Suffice? derives that the Bayes-optimal deferral rule depends on the difference between large- and small-model correctness minus cost. It shows that confidence thresholds fail in two settings: when the large model is a specialist that also fails out of its domain, and when inputs are noisy enough that neither model can solve them. That paper explicitly leaves open-ended generation untouched. Translation is a natural place to test it, because noisy sources and low-resource pairs where a compact specialist beats a generalist LLM are both common.
Formally, let $x$ be a source sentence and $d$ the small model's draft. For actions $a \in \{\text{keep}, \text{edit}, \text{retranslate}\}$ with expected cost $c_a$, the gate estimates $\hat{g}_a(x,d)$, the expected change in translation quality relative to keeping $d$. It then picks
$$ a^*(x,d) = \arg\max_a \; \hat{g}_a(x,d) - \lambda\, c_a , $$
where $\hat{g}_{\text{keep}} = 0$ by definition. The scalar $\lambda$ trades quality for cost and is set on held-out data to meet a budget. This is the multi-action, real-valued analogue of the headroom rule in When Does Confidence-Based Cascade Deferral Suffice?.
Relation to Prior Work
The proposal draws on three groups of work.
The first group motivates it. Translate Smart, not Hard establishes quality-estimation-gated translation cascades. Quality estimation here means a small model that scores a translation without a reference; the paper uses COMETKIWI and MetricX. It also reports that sentence length and model log-probabilities are unreliable gates. We keep its quality-estimation signal but change both what the gate predicts (gain, not quality) and what deferral does (repair or retranslate, not only retranslate). We also extend it beyond same-family pairs such as Tower-7B→70B to a heterogeneous pair: a compact encoder-decoder translator from Scaling neural machine translation to 200 languages escalating to a generative LLM.
The second group supplies the repair mechanism and constrains its design. SCALE shows that an LLM conditioned on a compact translator's draft and token confidences outperforms few-shot GPT-4 in low-resource settings. But SCALE calls the LLM on every sentence; our gate makes that call conditional. Leveraging GPT-4 for Automatic Translation Post-Editing shows that frontier post-editing improves strong WMT22 systems but sometimes hallucinates edits on acceptable translations. That is a direct argument for gating the edit action rather than applying it everywhere. Quality Estimation-Assisted Automatic Post-Editing shows that word-level OK/BAD tags focus a post-editor on real errors, reaching 71.1% edit precision on English–Marathi. Its "quality estimation as activator" setting is a binary threshold gate on a medium-sized editor. We reuse the idea of span-level guidance, give it to a black-box LLM through the prompt, and replace the fixed threshold with a gain predictor that competes against retranslation.
The third group supplies the cascade machinery. Gatekeeper assumes the large model dominates the small one and penalizes exact-token mismatch, and both assumptions fail in translation. We therefore do not retrain the small model; we leave that to future work. C3PO calibrates cascade thresholds with conformal prediction, a distribution-free method for setting thresholds with probability guarantees, to bound budget violations on reasoning tasks. We borrow its calibration of the cost side only, because our quality signal is continuous rather than exact-match. RouteLLM routes queries before any model runs. We gate after the draft exists, because in translation the draft and its error spans are the most informative signals available.
Research Program
The system has a small model, a large model, a quality estimator, and the gate.
We study two small/large pairings. The same-family pair, Tower-7B→Tower-70B, allows a direct comparison with Translate Smart, not Hard. The heterogeneous pair is the NLLB-200 3.3B compact translator escalating to an open generative LLM of roughly 70B parameters. The heterogeneous pair is where the large model is most likely to underperform the specialist on rare languages.
The quality estimator is COMETKIWI. It produces a sentence score and word-level error spans.
The three actions are defined as follows. "Keep" returns the draft. "Edit" prompts the large model with the source, the draft, and the flagged spans, and asks for a corrected translation. We also test a variant in which the model outputs only replacements for the flagged spans, which cuts generated tokens. "Retranslate" prompts the large model with the source alone.
The gate is a small regressor: a multilayer perceptron, plus a variant fine-tuned from the quality estimator's own encoder. Its inputs are the sentence quality score, error-span count and coverage, the small model's length-normalized log-probability and token entropy, source length, a language-pair embedding, and a source-noise feature from a source-side language-model perplexity. It outputs $\hat{g}_{\text{edit}}$ and $\hat{g}_{\text{retranslate}}$.
To train the gate, we run all three actions offline on a training pool drawn from earlier WMT test sets and FLORES-200 dev. We score each outcome with a reference-based metric, and the gain targets are differences between those scores. Training on one metric family and gating on another reduces the risk that the gate learns to please the quality estimator rather than to improve translations. Costs $c_a$ are measured in large-model FLOPs, which counts input and output tokens and therefore captures the longer edit prompt. We also measure wall-clock latency.
The value of $\lambda$ is chosen per budget on a calibration split. We report C3PO-style conformal bounds on realized cost so that a stated budget holds with a stated probability on new data. Our guarantees cover only the cost side.
Baselines
The naive reference points are the endpoints of the cost–quality curve. "All small" runs the small model only. "All large" runs the large model's direct translation on every sentence. We also include a random-deferral curve at matched rates, which shows how much any gate contributes beyond simply spending budget.
The strongest existing method is the quality-estimation-thresholded retranslation cascade of Translate Smart, not Hard, in both its budget-constrained and dynamic-threshold forms. It is the published system closest to ours and the one already shown to beat length and log-probability heuristics. For a fair comparison it gets the same small and large models, the same COMETKIWI estimator, the same test sets, and a threshold swept over the same budget grid. The only differences are our gate and our action space. Its reported result is that it matches full Tower-70B quality on WMT24 while deferring 30% to 50% of sentences. The file gives no per-language COMET or BLEU numbers ("no reported result found on Lacuna" for absolute scores), so we reproduce it ourselves. To beat it, our system must reach the all-large quality level at a significantly lower large-model FLOPs budget, or exceed it in quality at equal budget. We report paired bootstrap tests on the reference-based metric and on a human error-annotation subset.
The dense hybrid baseline is SCALE-style refinement applied to every sentence (SCALE). Its reported result is +5 COMET over few-shot GPT-4 in low-resource directions. It is an upper reference for the edit action's quality at maximum cost. We expect to match most of its quality at a fraction of its cost.
Three ablations isolate our ingredients. The first is a three-action gate driven by absolute quality-estimation bands with tuned cutoffs ("keep" above one cutoff, "edit" in the middle, "retranslate" below), which removes gain prediction. The second is a binary gain gate offering only keep or retranslate, which removes the edit action. The third is a two-way keep-or-edit gate, which corresponds to the "quality estimation as activator" setting of Quality Estimation-Assisted Automatic Post-Editing transplanted to an LLM editor.
Why not just…?
Why not just use the absolute quality-estimation threshold and retranslate, as already published? This baseline is strong. It is also possible that a gain predictor reduces to a monotone function of the quality score, since a low score both raises potential gain and drives deferral. If that happens, the gain gate gives no advantage on clean in-domain data. The experiment settles it by stratifying results. On clean high-resource WMT24 sentences we expect little difference. The claim stands or falls on the noisy and low-resource strata described below, where When Does Confidence-Based Cascade Deferral Suffice? predicts absolute thresholds will waste budget. If the gain gate does not win there either, we will report that headroom modeling adds nothing for translation.
Why not just post-edit every deferred sentence instead of choosing between edit and retranslate? Editing keeps correct parts of the draft, but it anchors the large model to drafts that are structurally broken. Editing everything also invites the over-correction and hallucinated edits documented in Leveraging GPT-4 for Automatic Translation Post-Editing and Quality Estimation-Assisted Automatic Post-Editing. The two-way keep-or-edit ablation measures this directly. If it matches the three-way gate, we will drop retranslation and report the simpler system.
Why not just run both models and let the quality estimator pick the better output? Selection after running both is close to an upper bound on what a two-way choice can achieve. But it pays for the large model on every sentence, which removes the point of cascading, as RouteLLM argues. We include it as an oracle-cost reference. It also lets us measure how much of its quality our gate recovers at our budget.
Proposed Experiments
We evaluate on the WMT24 test sets for the same-family pair, matching the setting of Translate Smart, not Hard. We use FLORES-200 low-resource and non-English directions for the heterogeneous pair, matching SCALE. We use WMT22 English–German and English–Chinese for edit quality against Leveraging GPT-4 for Automatic Translation Post-Editing. Quality is measured with reference-based COMET and MetricX, never with the gate's own estimator. We add human error-span annotation on 500 sentences per setting and TER against the draft, which measures unnecessary edits. Cost is measured as large-model FLOPs and latency, and each system is summarized by the area under its cost–quality curve.
Three checks are decisive. The first is the noisy-source stress test. We corrupt a fraction of WMT24 sources with typo and speech-recognition-style noise and measure the share of large-model budget spent on sentences where neither model improves. The gain gate should spend markedly less than the absolute-threshold cascade at equal quality. The second is the specialist trap. On FLORES-200 directions where NLLB beats the LLM, an absolute gate defers to the weaker model, and the gain gate should learn to keep the draft. The third is the value of the edit action. On sentences routed to "edit," we compare edit and retranslate outcomes head to head, and we report whether span-only output actually reduces cost or whether edit's benefit is purely in quality. We also report the case in which none of these effects appear, because that would show binary quality-gated retranslation is already sufficient for translation.
References
- Translate Smart, not Hard: Cascaded Translation Systems with Quality-Aware Deferral
- Statistical Phrase-Based Post-Editing
- Leveraging GPT-4 for Automatic Translation Post-Editing
- When Does Confidence-Based Cascade Deferral Suffice?
- Scaling neural machine translation to 200 languages
- SCALE: Synergized Collaboration of Asymmetric Language Translation Engines
- Quality Estimation-Assisted Automatic Post-Editing
- Gatekeeper: Improving Model Cascades Through Confidence Tuning
- C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning
- RouteLLM: Learning to Route LLMs from Preference Data