Multi-Hop Transformer for Document-Level Machine Translation

Long Zhang, Tong Zhang, Haibo Zhang, Baosong Yang, Wei Ye, Shikun Zhang

Abstract

Standard Neural Machine Translation (NMT) often struggles with document-level context, leading to errors in coreference and polysemy. The Multi-Hop Transformer (MHT) addresses these issues by mimicking a human-like "draft-editing" process. By iteratively reasoning over multiple previous sentences (hops) and utilizing target-side drafts to mitigate exposure bias, MHT achieves state-of-the-art performance across several translation benchmarks.

Introduction

Traditional NMT models typically process sentences in isolation. This context-agnostic approach fails when a sentence contains ambiguous terms or pronouns that depend on earlier discourse. While context-aware models exist, they often lack explicit inter-sentence reasoning and suffer from "exposure bias"—a mismatch between training on ground-truth context and inferring with generated (potentially erroneous) translations.

Methods: The Multi-Hop Architecture

The Multi-Hop Transformer (MHT) introduces an architecture designed for iterative refinement through several key components:

  1. Multi-Hop Reasoning: The model uses a series of "hops" to attend to previous sentences in chronological order. This process gradually updates the representation of the current sentence, simulating how a human translator might refine a draft based on context.
  2. Target-Side Drafts: To address exposure bias, MHT uses a pre-trained sentence-level NMT model to generate "drafts" of context sentences during training, rather than relying on ground-truth references.
  3. Context Gating: A learnable gating mechanism ($\alpha$) dynamically balances the information from the current sentence with the insights gathered from the multi-hop reasoning process.
The Multi-Hop Transformer architecture, featuring source-side and target-side sentence encoders and iterative attention layers.

Experimental Results

The MHT was evaluated on four diverse document-level translation tasks: TED Talks (En-De, Zh-En), Opensubtitles (En-Ru), and Europarl7 (En-De). These datasets provide a mix of formal and informal discourse, offering a robust testing ground for the model's reasoning capabilities.

Quantitative Performance

The MHT consistently outperformed the standard Transformer baseline and other context-aware models like CADec and CA-HAN. On average, it improved BLEU scores by 1.22 points over the baseline, achieving a notable 31.25 BLEU on the Europarl7 task.

Comparison of MHT performance against baseline and state-of-the-art context-aware models across multiple datasets.

Qualitative Analysis

The model demonstrated superior handling of discourse phenomena. For instance, in German translation, MHT correctly identified the neutral gender for "hum" (referring to an engine) where baselines failed. It also correctly translated polysemous words like "show" (as a TV program) by leveraging preceding context.

Key Insights

  • Reasoning Direction: Left-to-right reasoning (natural reading order) significantly outperformed right-to-left, highlighting the importance of logical document flow.
  • Context Window: Performance peaked at a window size of 3 sentences. Beyond this, additional context provided diminishing returns or introduced noise.
  • Draft Utility: Training with generated drafts proved more effective than using ground-truth references, confirming that reducing exposure bias is critical for real-world performance.

Conclusion and Future Work

The Multi-Hop Transformer provides a robust framework for document-level translation by explicitly modeling inter-sentence reasoning. While it introduces some computational overhead and depends on high-quality pre-trained models for drafting, its ability to resolve complex discourse issues marks a significant advancement. Future research may explore dynamic context windows to handle even longer documents more efficiently.

Summary updated

BibTeX
@inproceedings{zhang2021multi,
  author = {Long Zhang and Tong Zhang and Haibo Zhang and Baosong Yang and Wei Ye and Shikun Zhang},
  title = {{Multi-Hop Transformer for Document-Level Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.309},
}
Figure