Two-Way Coverage Rewards for Metric-Tuned Translation

Translation models are increasingly fine-tuned by reinforcement learning against an automatic quality score, such as COMET (a neural network trained to predict human quality judgments) or the agreement of a sample with other samples from the same model. These scores are one-sided. They mostly ask whether the output looks like a good translation. They do not separately ask whether every source word made it into the output, or whether every output word came from the source. A policy that drops a hard clause, or adds a fluent sentence nobody asked for, can still score well. We propose to split the training reward along the two directions of translation. A word aligner, a model that links each source word to the output words that translate it, produces two penalties: one for source content left unlinked (omissions) and one for output content left unlinked (hallucinations). Each penalty gets its own weight, and both are added to the usual metric reward during policy optimization.

The two research lines in the title meet here. Classical alignment research learned that a single direction of modeling misses errors the other direction would catch, and that forcing the two directions into hard agreement destroys legitimate many-to-many links. Research on the gap between training objectives and evaluation metrics learned to optimize non-differentiable scores directly, but it treats the score as a trusted scalar. Our claim is that the remaining training-evaluation gap in metric-tuned translation is directionally lopsided. Reward hacking under one-sided metrics shows up mainly as omissions and hallucinations. A cheap two-direction coverage signal, kept separate per direction rather than symmetrized, closes much of that gap without sacrificing the metric gains.

Core Claim and What Is New

The closest existing work is Consensus Group Relative Policy Optimization for Text Generation (C-GRPO). It samples a group of translations and rewards each by its agreement with its peers, distilling minimum Bayes risk decoding into a single-pass policy. Its consensus lives entirely among target-language candidates. The source sentence never enters the reward, so a consensus that all candidates share, such as everyone skipping the same hard phrase, is rewarded rather than penalized.

Noisy-Channel Minimum Bayes Risk Decoding supplies the conceptual seed. It shows that utility metrics treat hypothesis and reference asymmetrically, and that weighting the two channels separately improves candidate selection. It works only at inference time, and its asymmetry runs between hypothesis and reference, not between source and output.

OTTAWA supplies the detection method. It is an optimal-transport aligner with an adaptive NULL token, meaning a "nothing" slot that unaligned words can link to. It detects hallucinations and omissions on HalOmi, but it has only ever been used as a detector, never as a training signal.

What is new here is using source-side and output-side NULL mass as two separately weighted risk terms inside on-policy optimization. We also test whether the asymmetry matters, that is, whether the two directions need different weights.

Concretely, for source $x$ and sampled output $h$, an aligner computes a transport plan between the words of $x$ and $h$, each augmented with a NULL slot. Let $O(x,h)$ be the fraction of source mass sent to NULL and $H(x,h)$ the fraction of output mass sent to NULL. The reward is

$$R(x,h) = U(h) - \lambda_o\, O(x,h) - \lambda_h\, H(x,h),$$

where $U$ is the base utility (C-GRPO consensus or reference COMET), and $\lambda_o, \lambda_h \ge 0$ are separate weights for omissions and hallucinations. Group-relative advantages are computed on $R$ exactly as in C-GRPO.

Because the penalty acts on NULL mass rather than on link-by-link agreement between two directional models, it does not push alignments toward one-to-one links. This matters because Generalized Agreement for Bidirectional Word Alignment showed that hard agreement of that kind strips the many-to-many correspondences that idioms and morphologically rich languages need. The alignment-quality literature also constrains the design. Going Beyond AER and Discriminative Modeling of Extraction Sets found that link-level accuracy predicts downstream quality poorly. We therefore use only a coarse, coverage-level summary of the alignment and never reward link precision.

Research Program

The program has three stages, and the first can falsify the premise.

Stage 1: diagnosis. We measure whether metric-driven policy optimization actually increases omissions and hallucinations. We fine-tune a Llama-family translator on WMT En-Ja, En-Zh and En-De with C-GRPO and with COMET-reward GRPO. Checkpoints are scored on COMET and on omission and hallucination rates. These rates come from a detector held out from training: ALTI-style source attribution as used in HalOmi, plus a human-annotated sample. If error rates stay flat while COMET rises, the motivating gap does not exist at this scale, and the proposal should shrink to a negative result.

We found no reported omission or hallucination rates for RL-tuned translators on Lacuna. The closest evidence is indirect. Minimum Risk Training for Neural Machine Translation reports that risk-trained models degrade on sentences over 50 words, which is where omissions concentrate.

Stage 2: the aligner as reward. We use an OTTAWA-style aligner over frozen multilingual embeddings, kept frozen during policy training so the policy cannot co-adapt it. Before training, we check two things: its omission and hallucination detection on HalOmi, and its alignment error rate on the five-pair alignment benchmark used by AWESOME and Cross-Align.

Stage 3: training. We train with the two-penalty reward and sweep $\lambda_o$ and $\lambda_h$ independently. We report the trade-off frontier between COMET and error rate, broken down by sentence length and language pair. Japanese-English and Chinese-English are the most directionally lopsided pairs in the alignment tables, so we expect them to benefit most.

Baselines

The naive reference point is the base model before any reinforcement learning, decoded greedily. We also include its MLE-fine-tuned counterpart on the same parallel data. This tells us how much of any omission increase is caused by policy optimization itself.

The strongest existing method is C-GRPO as described in Consensus Group Relative Policy Optimization for Text Generation. It reports raising Llama En-Ja COMET on WMT24 from 0.530 to 0.759. We re-run it rather than copying numbers, using the same base model, group size, number of update steps and sampling temperature as our method. Its hyperparameter search gets the same budget we spend on $\lambda_o$ and $\lambda_h$.

We also run reference-COMET GRPO and sampled minimum risk training following Minimum Risk Training for Neural Machine Translation, both with matched compute. As an inference-time competitor, we run noisy-channel MBR from Noisy-Channel Minimum Bayes Risk Decoding on the base model's samples, with an equal sample budget.

Beating these baselines means the following. COMET must stay within the run-to-run variation of C-GRPO across seeds. At the same time, the held-out omission and hallucination rates must fall significantly, both overall and in the over-50-word bracket. Human judges on a blind sample must prefer our outputs for adequacy.

Three ablations isolate the key ingredient. The first uses an omission penalty only. The second uses a hallucination penalty only. The third ties the two weights together ($\lambda_o=\lambda_h$). If the tied version matches the untied one, the asymmetry claim fails, and we will report only the weaker claim that coverage helps.

Why not just…?

Why not just use an error-aware metric like XCOMET as the reward? This is the most serious threat. XCOMET-style metrics are trained to flag error spans, so they may already penalize omissions. We include XCOMET-reward GRPO as a baseline. If it achieves the same error reduction, the contribution reduces to "use a better metric." Two things could still favor our design. Learned metrics can themselves be gamed by the policy. The aligner, by contrast, is a separate frozen model whose penalties act directly on source coverage. The held-out detector and human evaluation will settle which approach actually covers more of the source.

Why not just symmetrize the metric, scoring both $U(h,r)$ and $U(r,h)$? Symmetrizing a target-side metric still never consults the source. A reference-free consensus reward like C-GRPO's has no reference to swap in at all. Noisy-channel MBR found that the best channel weights differ by metric, which argues against forcing equal treatment. The symmetrized-COMET reward is included as a baseline, and the tied-weight ablation tests the same intuition on our side.

Why not just add a length penalty? Omissions often shorten outputs, so a length-ratio penalty is a cheap proxy. However, it cannot distinguish dropping a clause from compressing wordy phrasing. It also cannot detect a hallucination that has the right length. We run a tuned length-ratio penalty with the same search budget. The decisive comparison is on HalOmi-style annotated errors whose length ratios are normal.

Why not filter or rerank at inference with OTTAWA instead of training? Reranking keeps the quadratic sampling cost that C-GRPO was designed to remove. It also cannot repair an omission that every sample shares. We include OTTAWA-penalized reranking of 64 samples as a baseline and report its latency alongside its quality.

Decisive Checks

The idea fails under any of three outcomes. First, Stage 1 shows no rise in omissions or hallucinations under C-GRPO. Second, XCOMET-reward GRPO or the length penalty matches our error reduction at equal COMET. Third, the error reduction appears only according to the training aligner and not according to the held-out ALTI detector and human annotators, which would indicate the policy learned to fool the aligner.

The idea succeeds if three things hold together. Separately weighted coverage penalties must lower held-out omission and hallucination rates at roughly unchanged COMET. The gain must be largest on long sentences and on Japanese-English and Chinese-English. The untied weights must beat the tied ones.