Proceedings of the Fourth Workshop on Neural Generation and Translation

Alexandra Birch, Andrew Finch, Kenneth Heafield, Hiroaki Hayashi, Yusuke, Ioannis Konstas, Graham Neubig, Xian Li, Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Gra- Ham, Kevin Lin, Ming-Yu Liu, Ming–Ting Sun, Jan Kautz, Michael Auli, David Grangier, Denis Yarats, Yann Dauphin, Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Dianqi Li, Yizhe Zhang, Zhe Gan, Yu Cheng, Chris Brockett, Bill Dolan, Jiwei Li, Michel Galley, Sp- Ithourakis Georgios, Jianfeng Gao, Matt Post, Ratish Puduppully, Li Dong, Mirella Lapata, Ilya Sutskever, Oriol Vinyals, Quoc V Le, Sam Wiseman, Stuart Shieber, Alexander Rush, Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, Yonatan Belinkov, Llus Mrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, James Glass, Alham Fikri, Aji, Sajid Anwar, Kyuyeon Hwang, Wonyong Sung, Antonio Valerio, Miceli Barone, Jindich Helcl, Rico Sennrich, Barry Haddow, En, Loc Barrault, Ondej Bojar, Marta Costa-Juss, Yvette Graham, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christos Christodouloupoulos, Mark Steedman, Chenhui Chu, Raj Dabre, Sadao Kurohashi, Rui Wang, Zi-Yi Dou, Keyi Yu, Antonios Anastasopoulos, Chelsea Finn, Pieter Abbeel, Sergey Levine, Jiatao Gu, Yong Wang, Yun Chen, O Victor, Kyunghyun Li, Xi'an LiOpenalex Unknown 2020 · DOI · OpenAlex

Abstract

The Fourth Workshop on Neural Generation and Translation (WNGT 2020) represents a pivotal moment in natural language processing, marking a shift from "accuracy at any cost" to "accuracy that fits in your pocket." This paper summarizes the research trends and results from three shared tasks: efficient neural machine translation (NMT), document-level generation and translation (DGT), and the STAPLE task for diverse translation generation. Key findings include the viability of 4-bit quantization, the discovery that deep encoders paired with shallow decoders drastically reduce latency, and the development of models capable of generating the diverse range of paraphrases necessary for language education.

Introduction: Addressing Practical Deployment Hurdles

While NMT has reached impressive quality levels, it faces three major hurdles for practical deployment:

  1. Computational Bloat: State-of-the-art models are massive, requiring expensive GPUs and significant memory, making them difficult to run on CPUs or mobile devices.
  2. Context Blindness: Most models translate sentence-by-sentence, losing the "thread" of a story or document, which leads to inconsistent pronouns and disjointed narratives.
  3. The "One Best Answer" Fallacy: Standard MT aims for a single "correct" output. However, in fields like language education (e.g., Duolingo), a system needs to recognize all valid ways to translate a prompt to provide meaningful feedback to students.

Methodologies for Efficient and Contextual NMT

The proceedings highlight several innovative technical approaches:

  • Knowledge Distillation: Using a massive "teacher" model to train a much smaller "student" model was a cornerstone for the Efficiency Task.
  • Quantization: Researchers pushed the boundaries of precision, moving from 32-bit floats to 8-bit and even 4-bit logarithmic quantization. By leaving bias terms uncompressed, they achieved massive size reductions with negligible quality loss.
  • Architectural Asymmetry: A recurring theme was using deep encoders (up to 35 layers) but very shallow decoders (often just 1 layer). Since the decoder is the bottleneck during inference, this shift provides a massive speedup.
  • Consensus Voting and Filtering: For the STAPLE task, teams used "n-best" list generation followed by binary classifiers to filter out translations that, while technically correct, didn't sound like something a human learner would actually say.

Shared Tasks and Datasets

The workshop was structured around three primary shared tasks:

  • Efficiency Task: Utilized the standard WMT 2019 English-German benchmarks to test speed and size.
  • Document-level Generation and Translation (DGT): Used the Rotowire dataset, requiring systems to generate descriptive summaries from NBA basketball box-score tables.
  • STAPLE Task: Introduced a novel dataset from Duolingo containing English prompts paired with comprehensive, weighted lists of acceptable human translations in five languages. Weights reflect "Learner Response Frequency" (LRF).

Key Insights and Results

The Decoder Bottleneck and Quantization

In the Efficiency Task, the "aha moment" was realizing that the decoder's autoregressive nature is where the time is lost. Pruning the decoder while keeping the encoder deep is the most effective way to save time without losing understanding. Furthermore, NMT model weights are skewed toward zero; logarithmic quantization captures this distribution much better than standard fixed-point methods, allowing models to shrink by 8x while staying within 0.2 BLEU points of the original.

BLEU vs. Diversity

A major finding in the STAPLE task was that models with the highest BLEU scores often performed poorly on the task's actual metric (Weighted F1). This suggests that training for the "average" translation actually hurts a model's ability to generate the "long tail" of valid paraphrases.

Performance Benchmarks

  • Speed: The fastest systems could translate 1 million lines in under 3 seconds on a single GPU. On CPU, the OpenNMT team achieved a 13x speedup over their baseline with only a 0.8 BLEU point drop.
  • Size: Models were successfully compressed from ~250MB down to under 20MB using 4-bit quantization.
  • STAPLE Performance: The top-performing team (JHU) swept all five language tracks, outperforming the Amazon Translate baseline by factors of 2 to 10 in weighted F1 score.
  • Document Context: The FJWU team showed that hierarchical attention (looking at previous sentences) improved translation quality by over 2 BLEU points compared to sentence-level baselines.

Limitations and Future Work

Despite the progress, several challenges remain. While 4-bit quantization saves disk space, current hardware doesn't natively support 4-bit math instructions, meaning speed gains aren't yet as large as size gains. Systems still struggle with highly inflected languages like Hungarian, and the DGT task had low participation, highlighting the difficulty of finding document-level parallel data. Future directions include moving toward reinforcement learning with semantic rewards (like XSIM) rather than relying on simple word-overlap metrics like ROUGE or BLEU.

Summary updated

Abstract

We describe the finding of the Fourth Workshop on Neural Generation and Translation, held in concert with the annual conference of the Association for Computational Linguistics (ACL 2020). First, we summarize the research trends of papers presented in the proceedings. Second, we describe the results of the three shared tasks 1) efficient neural machine translation (NMT) where participants were tasked with creating NMT systems that are both accurate and efficient, and 2) document-level generation and translation (DGT) where participants were tasked with developing systems that generate summaries from structured data, potentially with assistance from text in another language and 3) STAPLE task: creation of as many possible translations of a given input text. This last shared task was organised by Duolingo.
BibTeX
@inproceedings{birch2020proceedings,
  author = {Alexandra Birch and Andrew Finch and Kenneth Heafield and Hiroaki Hayashi and Yusuke and Ioannis Konstas and Graham Neubig and Xian Li and Loïc Barrault and Ondřej Bojar and Marta R. Costa-jussà and Christian Federmann and Mark Fishel and Yvette Gra- Ham and Kevin Lin and Ming-Yu Liu and Ming–Ting Sun and Jan Kautz and Michael Auli and David Grangier and Denis Yarats and Yann Dauphin and Ian Goodfellow and Jean Pouget-Abadie and Mehdi Mirza and Bing Xu and David Warde-Farley and Dianqi Li and Yizhe Zhang and Zhe Gan and Yu Cheng and Chris Brockett and Bill Dolan and Jiwei Li and Michel Galley and Sp- Ithourakis Georgios and Jianfeng Gao and Sp- Ithourakis Georgios and Jianfeng Gao and Matt Post and Ratish Puduppully and Li Dong and Mirella Lapata and Ilya Sutskever and Oriol Vinyals and Quoc V Le and Sam Wiseman and Stuart Shieber and Alexander Rush and Dzmitry Bahdanau and Kyunghyun Cho and Yoshua Bengio and Yonatan Belinkov and Llus Mrquez and Hassan Sajjad and Nadir Durrani and Fahim Dalvi and James Glass and Alham Fikri and Aji and Kenneth Heafield and Sajid Anwar and Kyuyeon Hwang and Wonyong Sung and Antonio Valerio and Miceli Barone and Jindich Helcl and Rico Sennrich and Barry Haddow and Alexandra Birch and En and Loc Barrault and Ondej Bojar and Marta Costa-Juss and Christian Federmann and Mark Fishel and Yvette Graham and Barry Haddow and Matthias Huck and Philipp Koehn and Shervin Malmasi and Christos Christodouloupoulos and Mark Steedman and Chenhui Chu and Raj Dabre and Sadao Kurohashi and Chenhui Chu and Rui Wang and Zi-Yi Dou and Keyi Yu and Antonios Anastasopoulos and Chelsea Finn and Pieter Abbeel and Sergey Levine and Jiatao Gu and Yong Wang and Yun Chen and O Victor and Kyunghyun Li and Cho and Xi'an Li},
  title = {{Proceedings of the Fourth Workshop on Neural Generation and Translation}},
  booktitle = {Openalex Unknown},
  year = {2020},
  doi = {10.18653/v1/2020.ngt-1},
}
Figure