Improving Document-Level Neural Machine Translation with Domain Adaptation

Sami Ul Haq, Sadaf Abdul Rauf, Arslan Shoukat, Noor-e-HiraACL 2020

TL;DR

This paper tackles the challenge of improving neural machine translation (NMT) by incorporating document-level context, which traditional sentence-by-sentence models miss. They demonstrate that using previous sentences as context significantly boosts translation quality, and further enhance performance by adapting models to specific domains using techniques like corpus replication.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) has revolutionized translation, but standard models operate on individual sentences. This approach ignores the broader context of a document, leading to translations that can be incoherent or miss subtle nuances that span across sentences. While document-level NMT aims to fix this, a major hurdle is the scarcity of large, parallel document-level corpora. This scarcity makes it difficult to train models that perform well across different domains. The paper addresses this by exploring how to effectively adapt document-level NMT models to new domains, even with limited in-domain data.

Methods

The researchers employ the Transformer architecture, a standard in NMT, and extend it for document-level translation. Their core innovation lies in integrating contextual information from previous sentences using hierarchical attention networks (HAN). They explore two main model types:

  • Sentence-level Models: A baseline Transformer model trained on individual sentences.
  • Document-level Models: These models incorporate context from previous sentences. They use a hierarchical attention encoder and decoder to capture both source and target side contextual information.

For domain adaptation, they investigate two primary strategies:

  • Corpus Replication: Over-sampling the in-domain corpus by replicating it multiple times during training. This forces the model to pay more attention to the specific domain.
  • Fine-tuning: While not explicitly detailed as a separate experiment, the paper discusses fine-tuning as a common domain adaptation technique.

Datasets

The study utilizes several datasets for training and evaluation, primarily focusing on English-German translation:

  • In-domain Data:
    • RotoWire: A sports-related dataset consisting of NBA game summaries, treated as individual documents.
  • Out-of-domain Data:
    • Europarl v9: A parliamentary proceedings corpus.
  • Rapid: A corpus likely containing news or general text.
  • News Commentary v14: A news commentary corpus.

The paper notes that for Europarl v9, they redefined document boundaries to ensure a more manageable context size, averaging 14 sentences per document.

Key Insights and Ideas

  • Context is King (Even for NMT): The paper strongly reiterates that incorporating document-level context, even just a few previous sentences, significantly improves translation quality over standard sentence-level NMT.
  • Domain Adaptation is Crucial: NMT models trained on general domains often perform poorly on specialized domains. This paper shows that domain adaptation techniques, particularly corpus replication, can effectively bridge this gap.
  • Source-side Context Matters More: The results suggest that providing contextual information on the source side (encoder) is more impactful than on the target side (decoder) for improving translation quality.
  • Simplicity Can Be Effective: Over-sampling the in-domain data (corpus replication) proved to be a straightforward yet highly effective method for domain adaptation, without requiring architectural changes.

Key Results

  • Document-level vs. Sentence-level: Document-level models achieved significant improvements in BLEU scores compared to the sentence-level baseline. For instance, a baseline Transformer model achieved 32.18 BLEU, while incorporating HAN encoder and decoder boosted it to 38.32 BLEU.
  • Domain Adaptation Impact:
    • On sentence-level models, adding a small amount of in-domain data (RotoWire) to a larger out-of-domain corpus (Rapid) resulted in a substantial +4 BLEU point improvement.
  • Replicating the in-domain RotoWire corpus twice and thrice led to further significant gains, with the best sentence-level model achieving 20.01 BLEU points.
  • For document-level models, replicating the RotoWire corpus thrice (RotoWire(*3) + Rapid + NC + Euro) yielded the best score of 43.08 BLEU.
  • Contextual Experimentation: Experiments with regular, random, and no context demonstrated that models trained with consistent contextual information performed best, and their performance degraded when context was randomized or removed.

Limitations and Future Work

  • Limited Context Window: While HAN models are dynamic, the paper notes that their approach, like others, is somewhat limited by the number of previous sentences considered for context (they used k=3).
  • Decoder Context Contribution: The joint HAN encoder-decoder model did not perform as well as expected, potentially because the HAN decoder wasn't providing complementary information. The authors suggest exploring other configurations for decoder context.
  • Data Scarcity: The fundamental challenge of limited document-level parallel data remains.
  • Future Work: The authors suggest exploring other domain adaptation techniques and further investigating the role of source vs. target context. They also mention that their best model (HAN encoder with domain adaptation) was submitted for official evaluation.

Summary updated

BibTeX
@inproceedings{haq2020improving,
  author = {Sami Ul Haq and Sadaf Abdul Rauf and Arslan Shoukat and Noor-e-Hira},
  title = {{Improving Document-Level Neural Machine Translation with Domain Adaptation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure