Language Model Based Target Token Importance Rescaling for Simultaneous Neural Machine Translation

Aditi Jain, Nishant Kambhatla, Anoop SarkarACL 2023

TL;DR

This paper tackles the challenge of simultaneous machine translation (SiMT), where real-time translation quality is often sacrificed for speed. They introduce a novel method that uses an auxiliary target-side language model to dynamically rescale the importance of target tokens during training. This approach significantly improves translation quality, especially for rare words and longer sentences, while maintaining low latency.

What Problem Does This Paper Address?

Simultaneous Machine Translation (SiMT) aims to translate text in real-time as it's being received, creating a fundamental trade-off between translation quality and latency. Traditional NMT models, when adapted for SiMT, struggle to balance these competing demands. Existing adaptive policies, which decide whether to read more source tokens or write a target token, often fail to account for the varying importance of different target tokens. This can lead to suboptimal translations, particularly when dealing with rare words or complex sentence structures, as the model might treat all tokens equally in its decision-making process. The paper aims to bridge this gap by making the translation process more "aware" of the importance of each token it generates.

Methods

The core of this paper's contribution lies in its "Target Adaptive Training" approach, augmented with an auxiliary target-side language model (LM). Instead of relying solely on the standard cross-entropy loss for training the decoder, they introduce a mechanism to rescale the importance of each target token. This rescaling is guided by the probability assigned to a target token by an independent language model, relative to the probability assigned by the main translation model.

Specifically, they define a "Target Information Quotient" (TIQ) that quantifies the importance of a target token given the source context and the already generated target prefix. This TIQ is then used to derive token-level and sentence-level weights. These weights are incorporated into the training objective, effectively re-weighting the loss for each token. Tokens deemed more "informative" by the LM (and thus potentially more difficult or important for the translation) receive higher weights, encouraging the model to pay more attention to them. This method builds upon the Monotonic Multihead Attention (MMA) framework, enhancing its adaptive policy.

Datasets

The experiments were conducted on the following datasets:

  • IWSLT'15 English ↔ Vietnamese: 133K training pairs, with TED tst2012 for validation and TED tst2013 for testing.
  • IWSLT'14 English ↔ German: 160K training pairs, with validation and test sets of 7283 and 6750 pairs, respectively.

The paper does not introduce new benchmarks but evaluates its method on established SiMT datasets.

Key Insights and Ideas

  • Target Token Importance Matters: The paper strongly suggests that not all target tokens are created equal in SiMT. Prioritizing the generation of more informative or rare tokens can significantly boost translation quality.
  • Auxiliary LM as a "Quality Signal": An independent language model can provide a valuable signal about the "naturalness" or expectedness of a target token, which can be leveraged to guide the translation model's focus.
  • Low Latency Doesn't Mean Low Quality: The proposed method demonstrates that it's possible to achieve high translation quality even at very low average lagging values (AL), challenging the notion that speed inherently degrades quality.
  • Rare Words Benefit Most: The re-weighting mechanism proves particularly effective for translating rare words, which are often a bottleneck in translation quality.
  • Improved Read/Write Path: The method leads to a more correlated read/write path with source-target alignments, suggesting a more efficient and informed decision-making process.

Key Results

The proposed MMA+TC (MMA with Target Context) method consistently outperforms several state-of-the-art baselines across various language pairs (Vi→En, En→De, De→En) in terms of BLEU scores at comparable low latencies.

  • Significant BLEU Improvement: For Vi→En, the method achieves offline translation quality at an Average Lagging (AL) of 3.86.
  • Low Latency Performance: The method shows notable improvements in BLEU scores at lower latencies, where source context is more limited.
  • Rare Word Translation: The F-measure for rare words (frequency ≤ 10) is improved, and the model better captures tokens from lower frequency bins compared to the baseline MMA.
  • Translation Length: The model shows a significant improvement in BLEU for longer sentences and tends to produce translations closer to the ground truth length, avoiding over-generation.
  • Sufficiency of READ Actions: The method demonstrates higher "sufficiency" in its READ actions, meaning it reads just enough source tokens to inform the translation of a target token, leading to better alignment with reference paths.

Limitations and Future Work

  • Model Size and Training Time: The auxiliary target-side LM increases the model size and training time, which can be a bottleneck for very large translation models.
  • Suboptimal MMA Policy: While the proposed method improves upon MMA, MMA itself has limitations due to its multi-head decision-making, which could be further optimized.
  • Future Work: The authors suggest exploring more efficient ways to incorporate target context, optimizing the policy to follow reference alignments more closely for better latency, and investigating the use of additional monolingual data to strengthen the language model.

Summary updated

BibTeX
@inproceedings{jain2023language,
  author = {Aditi Jain and Nishant Kambhatla and Anoop Sarkar},
  title = {{Language Model Based Target Token Importance Rescaling for Simultaneous Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure