Cheat Codes to Quantify Missing Source Information in Neural Machine Translation

Proyag Pal, Kenneth HeafieldNAACL 2022 · DOI

Introduction

In neural machine translation (NMT), the "one-to-many" problem remains a fundamental challenge: a single source sentence can be correctly translated into a target language in numerous ways, varying in style, tone, and lexical choice. However, standard NMT models are typically evaluated on their ability to recreate a single, specific reference translation. This creates an "information gap"—formally $H(t|s)$—representing the additional information required to bridge the distance between the source sentence and a specific reference. Researchers at the University of Edinburgh have introduced a novel method to quantify this gap by providing models with highly compressed "cheat codes" derived from the target text.

Methodology

The core of this approach is a modified dual-encoder Transformer architecture. As illustrated in the architecture diagram below, the system utilizes two distinct paths to inform the decoder.

Figure 1: The dual-encoder architecture featuring a Transformer for the source and a GRU-based bottleneck for the reference "cheat code."
  1. Source Encoding: A standard Transformer encoder processes the source sentence.
  2. Reference Encoding (The Cheat Code): A second encoder, using a Gated Recurrent Unit (GRU), processes the reference translation. This output is passed through a linear bottleneck to create the "cheat code"—a compressed representation of the target's unique nuances.
  3. Decoding: The Transformer decoder receives both the source encoding and the cheat code to generate the final output.

The researchers experimented with different bottleneck capacities, including fixed-length vectors for the entire sentence and variable-length vectors (floats per token). They also tested quantization limits, reducing precision to as low as 4 bits to find the absolute lower bound of useful information.

Key Results and Insights

The study, conducted on the WMT21 German→English dataset, revealed that NMT models are incredibly sensitive to even small amounts of external guidance. The following table summarizes how different cheat code configurations impact translation quality across various metrics.

Figure 3: Performance metrics across various cheat code configurations, showing the transition from baseline to near-perfect reproduction.

The Threshold for Perfection

As shown in Figure 3, the standard Transformer baseline achieves a BLEU score of 32.2 on Reference A. By providing a variable-length cheat code of just one 32-bit float per token, the score jumps to 40.1. Remarkably, at two floats per token, the model achieves "near-perfect" reproduction with a BLEU score of 92.4. This suggests that approximately 64 bits of information per token is the upper bound of the information missing from the source required to perfectly match a specific human reference.

Metric Blind Spots

A significant finding involved COMET-QE, a reference-less evaluation metric. While traditional metrics like BLEU and ChrF skyrocketed as cheat code capacity increased, COMET-QE scores remained low. This suggests that reference-less metrics may be tuned to identify "typical" machine translation artifacts and struggle to accurately evaluate outputs that are "too perfect" or mimic human ground truth too closely.

Semantic Interpolation

The researchers also discovered that these cheat codes exist in a continuous semantic space. By mathematically interpolating between the cheat codes for two different references (Reference A and Reference B), the model can generate hybrid translations that blend the styles of both inputs.

Conclusion and Future Work

This work provides a concrete method for quantifying the information gap in translation. While the study successfully measures how much information is missing (in bits or floats), future research is needed to qualify that information—determining whether it represents stylistic choices, cultural nuances, or specific terminology. These findings pave the way for "steerable" NMT systems where users could potentially use low-dimensional codes to guide a model toward a desired translation style or level of formality.

Summary updated

BibTeX
@inproceedings{pal2022cheat,
  author = {Proyag Pal and Kenneth Heafield},
  title = {{Cheat Codes to Quantify Missing Source Information in Neural Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.177},
}
Figure