Findings of the Third Workshop on Neural Generation and Translation.

Hiroaki Hayashi, Yusuke Oda, Alexandra Birch, Ioannis Konstas, Andrew M. Finch, Minh-Thang Luong, Graham Neubig, Katsuhito SudoharXiv 2019 · OpenReview

Abstract

The WNGT 2019 workshop findings highlight a significant shift in the NLP community from sentence-level translation toward complex document-level generation and computational efficiency. This paper summarizes the results of two shared tasks: (1) Efficient Neural Machine Translation (NMT), where participants optimized for both accuracy and speed, and (2) Document-level Generation and Translation (DGT), which focused on generating multi-sentence narratives from structured data. Key findings demonstrate that techniques like 8-bit quantization and student-teacher distillation allow highly optimized systems to achieve state-of-the-art accuracy with significantly reduced latency.

Introduction: Document-Level Coherence and Efficiency

While neural sequence-to-sequence models are the current standard in NLP, two major gaps persist. First, most research focuses on isolated sentences, whereas real-world applications—like sports reporting or dialogue—require document-level coherence and the ability to synthesize information from structured data (e.g., box scores). Second, as models grow in complexity to improve accuracy, they often become too computationally expensive for production environments. The workshop addressed these by hosting the DGT and Efficient NMT shared tasks, challenging researchers to find the "Pareto frontier" where translation quality meets hardware efficiency.

Methods and Optimization Techniques

Participants employed several cutting-edge techniques across the two tasks:

  • Document-level Context: Teams like FIT-Monash used hierarchical and flat attention mechanisms to allow models to consider the entire document context rather than just the current sentence.
  • Data Augmentation: Extensive use of back-translation (translating target-language text back into the source language to create more training data) was a common theme for improving document-level fluency.
  • Efficiency Optimizations: The Marian team implemented 8-bit matrix multiplication and parameter tying between decoder layers to reduce memory usage and improve CPU cache locality.
  • Model Compression: Team Notre Dame utilized "Auto-sizing" via block-sparse regularization to prune unnecessary rows and columns from weight matrices, effectively shrinking the model size.
  • Generation Strategies: Microsoft explored nucleus sampling to improve the diversity and quality of generated summaries, finding that evaluation metrics are highly sensitive to sampling hyperparameters (Top-$p$).

Datasets

  • RotoWire English-German: A newly introduced dataset for the DGT task, consisting of NBA basketball box scores paired with professional news articles. This provides a unique testbed for data-to-text generation in a multilingual setting.
  • WMT 2014/2019 English-German: Used as the primary training and evaluation data for the translation-heavy components and the efficiency task.
  • Newstest2015: The standard benchmark used to measure the final accuracy-vs-speed trade-offs in the efficiency task.

Key Insights

  • The "Generation" Pivot: For the first time, the workshop saw more papers focused on language generation (22) than on pure translation (14), signaling a broadening of the field's interests.
  • Pre-training is Robust: Pre-training models on massive news corpora proved effective even when the final task involved very different input types, such as linearized data tables.
  • Hardware-Aware Design: The success of the Marian team showed that software-level optimizations, such as custom C++ kernels for 8-bit math, can be just as impactful as algorithmic changes for real-world performance.

Key Results

Efficient NMT

In the CPU-based translation task, the Marian team dominated the Pareto frontier, providing systems that were both the most accurate and among the fastest. Their optimized Transformer models achieved BLEU scores over 30 on the newstest2015 set with significantly lower latency than standard baselines. The workshop also provided a rare look at peak memory consumption, noting that while some systems reserve large blocks of VRAM upfront, others maintain a minimal footprint.

Pareto frontier plots for the Efficient NMT task, showing accuracy (BLEU %) vs. Time and Memory for both CPU and GPU.

Document Generation

In the DGT task, Team Naver Labs Europe (NLE) and Team Microsoft outperformed the strong baselines. NLE’s approach showed that document-level fine-tuning is essential; models aware of document boundaries consistently beat sentence-level models.

Conclusion and Future Work

The authors acknowledge a persistent lack of document-level parallel data, which necessitated reliance on translated versions of existing datasets. Furthermore, while automated metrics like BLEU and ROUGE are standard, they don't fully capture "content fidelity"—whether the model accurately reflects the underlying data. Future work will focus on better evaluation metrics for document-level coherence and further integration of structured data with cross-lingual summarization.

Summary updated

Abstract

This document describes the findings of the Third Workshop on Neural Generation and Translation, held in concert with the annual conference of the Empirical Methods in Natural Language Processing (EMNLP 2019). First, we summarize the research trends of papers presented in the proceedings. Second, we describe the results of the two shared tasks 1) efficient neural machine translation (NMT) where participants were tasked with creating NMT systems that are both accurate and efficient, and 2) document-level generation and translation (DGT) where participants were tasked with developing systems that generate summaries from structured data, potentially with assistance from text in another language.
BibTeX
@misc{hayashi2019findings,
  author = {Hiroaki Hayashi and Yusuke Oda and Alexandra Birch and Ioannis Konstas and Andrew M. Finch and Minh-Thang Luong and Graham Neubig and Katsuhito Sudoh},
  title = {{Findings of the Third Workshop on Neural Generation and Translation.}},
  year = {2019},
  eprint = {1910.13299},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.13299},
}
Figure