When training or evaluating a machine translation system, it is tempting to rely on a single, standardized numerical score like BLEU to measure progress. However, optimizing a model exclusively for one metric introduces a significant vulnerability: the model quickly learns to exploit that metric's mathematical blind spots. Over-indexing on a single proxy metric can cause translation quality to stagnate or even degrade across other equally important dimensions of quality.
To build translation systems that are truly robust, fluent, and practically useful, researchers have shifted toward multi-metric optimization. This approach treats translation evaluation not as a single scalar problem, but as a multi-objective balancing act.
The Pitfall of Single-Metric Overfitting
Why does optimizing for a single metric fail? Every automated evaluation metric is a narrow proxy for human judgment. For instance, surface-level metrics like BLEU measure exact $n$-gram overlaps between a model's output and a human reference translation. If a system is tuned aggressively to maximize this overlap, it may learn to avoid valid synonyms, penalize legitimate reorderings, or produce grammatically awkward sentences that happen to match the reference text's exact phrasing.
This phenomenon was documented early in the development of statistical machine translation. In The CMU statistical machine translation system for IWSLT 2005, Alex Waibel et al. observed that optimizing a translation system solely for BLEU led to a distinct degradation in performance when evaluated on other metrics like NIST. Similarly, Adrià de Gispert et al. demonstrated in The TALP ngram-based SMT system for IWSLT'05 that maximizing BLEU on a development set failed to guarantee improvements in alternative metrics like mWER on the test set.
When an optimization algorithm targets only one metric, it treats the metric's blind spots as paths of least resistance, boosting the score without improving underlying translation quality.
The Divergent Dimensions of Quality
Translation quality is multi-dimensional, and different metrics capture fundamentally different aspects of what makes a translation "good." When we look closely at these metrics, we find several inherent trade-offs:
Surface Overlap vs. Post-Editing Effort
A translation can be highly intelligible to a human reader while still requiring significant structural modifications from a professional editor. In Joint WMT 2012 Submission of the QUAERO Project, Markus Freitag et al. showed that optimizing translation ensembles for word overlap (BLEU) versus edit distance (Translation Edit Rate, or TER) resulted in entirely divergent system configurations. A system tuned to excel at word overlap often performed poorly when measured by the actual physical effort required for a human to correct the output. As Luisa Bentivogli et al. detailed in Machine Translation Human Evaluation: an investigation of evaluation based on Post-Editing and its relation with Direct Assessment, there is a fundamental tension between measuring general linguistic adequacy (how well a reader understands the text) and practical utility for professional translation workflows.
Accuracy vs. Fluency (Simpson's Paradox)
At a macro level across an entire evaluation corpus, a system's accuracy and fluency scores typically appear highly correlated. However, Charles Kemp et al. revealed a striking statistical reversal in Simpson's Paradox and the Accuracy-Fluency Tradeoff in Translation. The authors demonstrated that at the level of individual sentences, accuracy and fluency frequently trade off. Choosing a highly accurate, literal word choice can disrupt local grammatical fluency, while smoothing out a phrase to make it sound natural can dilute its precise meaning.
Corpus Level: [Higher Accuracy] ──(Positively Correlated)──> [Higher Fluency]
Sentence Level: [More Accurate Word] <──(Inherent Trade-off)──> [Smoother/Fluent Phrase]
Segment-Level vs. System-Level Scale
Modern learned evaluation metrics (such as neural regression models) face a scale-dependent conflict. In MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task, Juraj Juraska et al. noted a persistent tension when training quality estimators: optimizing a model to accurately score individual sentences (segment-level) often degrades its ability to reliably rank overall document or system performance (system-level).
How Multi-Metric Optimization Works
To resolve these conflicts, modern translation frameworks integrate multiple metrics directly into system tuning and hypothesis selection. By optimizing for a combination of complementary criteria, these systems prevent the model from gaming any single metric.
Joint Metric Tuning
Instead of treating a single metric as the sole source of truth during training, systems can be optimized against a blended objective function. Christophe Servan et al. implemented this in Optimising Multiple Metrics with MERT by extending the Moses toolkit to optimize for a combination of BLEU and TER simultaneously. The authors proved that tuning for both metrics yielded better overall translation quality than tuning for BLEU alone, and surprisingly, it even improved the individual metric scores themselves.
By forcing the optimization process to satisfy two distinct mathematical formulations—one based on precision ($n$-grams) and one based on edit operations (TER)—the system regularizes itself. This multi-objective approach was also successfully leveraged by Matthias Huck et al. in The RWTH Aachen Machine Translation System for WMT 2011 to mitigate the biases inherent in individual evaluation scores, resulting in more balanced and human-readable outputs.
Multi-Stage Reranking
An alternative way to balance metric trade-offs is to separate generation from evaluation using a multi-stage pipeline. In NAIST-NICT WMT'23 General MT Task Submission, Hiroyuki Deguchi et al. addressed the trade-off between translation quality and diversity by generating hundreds of diverse candidate translations using varied decoding strategies.
Instead of relying on a single metric to pick the winner, the authors passed the candidates through a two-stage reranking system that combined surface-level overlap criteria with deep neural semantic metrics. This multi-stage alignment ensures that the final selection satisfies both literal accuracy and deeper semantic coherence, achieving a level of robustness that a single metric cannot provide.
Key Foundational Work
The CMU statistical machine translation system for IWSLT 2005 (2005) — contributed 3 concept elements to this direction.
The TALP ngram-based SMT system for IWSLT'05 (2005) — contributed 2 concept elements to this direction.
Optimising Multiple Metrics with MERT (2011) — contributed 1 concept elements to this direction.