On Search Strategies for Document-Level Neural Machine Translation

Christian Herold, Hermann NeyACL 2023 · DOI

Abstract

While research in document-level Neural Machine Translation (NMT) usually focuses on complex model architectures, this study shifts the spotlight to decoding strategies—the actual search process used to generate translations. By evaluating various search processes, we find that most common decoding methods perform surprisingly similarly on standard metrics. However, an "oracle" experiment reveals that the quality of context provided during decoding is a significant bottleneck for resolving tricky linguistic features like pronouns, suggesting that higher-quality context could significantly boost translation accuracy.

Introduction

Standard NMT often translates sentences in isolation, leading to "document-level" errors such as inconsistent gender, clashing styles, or ambiguous word choices that can only be resolved by considering neighboring sentences. To address this, researchers have developed "context-aware" models. However, there is no consensus on the best search (decoding) strategy for these models. Should the system translate the whole document at once, sentence by sentence, or via a multi-pass approach? This paper investigates whether improvements in document-level NMT are driven by model architecture or specific search settings.

Methods

The authors evaluate several search strategies using a standard "concatenation" model, where the current sentence and its context are fed into a Transformer together:

  • Full Segment: Translating non-overlapping blocks of sentences.
  • Last/First Sentence: Translating overlapping windows but only keeping the last (or first) generated sentence.
  • 2-Pass Decoding: Generating a sentence-level translation first, then using it as context for a second, document-aware pass.
  • Doc-Trans: Translating sentence-by-sentence, using previously generated hypotheses as context.
  • Doc-Trans (Beam): A version of Doc-Trans that maintains a beam of multiple document-level candidates.
  • Cheating (Oracle): Using actual ground-truth reference translations as context for analysis.

These strategies were tested on three standard benchmarks: NEWS (En→De), TED (En→It), and OS (En→De), as well as the ContraPro test set. Computationally, methods like doc-trans and full segment are efficient ($O(NL)$), whereas sliding window approaches like last sentence incur higher costs ($O(NLk)$).

Results

Search Strategy Indifference

For standard metrics like BLEU and TER, the specific decoding strategy had minimal impact. As shown in Figure 1, most methods yielded nearly identical scores across the different datasets. On the OS task, for example, document-level strategies achieved BLEU scores between 37.4 and 38.3, compared to 37.1 for the sentence-level baseline.

BLEU and TER scores across different datasets and decoding strategies.

The Quality Gap in Pronoun Resolution

While general metrics showed little variation, targeted evaluation of pronoun accuracy (using the ContraPro set) revealed a significant "Quality Gap." Document-level strategies improved pronoun F1 scores from a baseline of 41.4% to approximately 66% on the OS dataset. However, the "Cheating" oracle reached 73.7% F1, as illustrated in Figure 2. This suggests that the model's ability to use context is not the primary bottleneck; rather, it is the quality of the context generated during decoding.

Pronoun accuracy and style consistency across decoding strategies.

Context Directionality and Perplexity

The "First Sentence" strategy, which utilizes future context, performed poorly on pronoun resolution, confirming that past (left-side) context is more critical for anaphora. Additionally, the researchers observed a "Perplexity Paradox": ground-truth references often had higher perplexity than model hypotheses, indicating that models tend to prefer "safe," high-probability sequences over the linguistic variety found in human references.

Discussion and Conclusion

The study concludes that for general translation quality, simple and efficient decoding strategies like doc-trans or full segment are sufficient. However, the significant performance jump seen with the oracle context highlights a major opportunity for future research: improving the quality of context used during decoding. Future work might explore iterative refinement or better re-ranking mechanisms to bridge the gap between current decoding outputs and the potential shown by gold-standard context.

Summary updated

BibTeX
@inproceedings{herold2023search,
  author = {Christian Herold and Hermann Ney},
  title = {{On Search Strategies for Document-Level Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.811},
}
Figure