Abstract
The Fourth Workshop on Neural Generation and Translation (WNGT 2020) represents a pivotal moment in natural language processing, marking a shift from "accuracy at any cost" to "accuracy that fits in your pocket." This paper summarizes the research trends and results from three shared tasks: efficient neural machine translation (NMT), document-level generation and translation (DGT), and the STAPLE task for diverse translation generation. Key findings include the viability of 4-bit quantization, the discovery that deep encoders paired with shallow decoders drastically reduce latency, and the development of models capable of generating the diverse range of paraphrases necessary for language education.
Introduction: Addressing Practical Deployment Hurdles
While NMT has reached impressive quality levels, it faces three major hurdles for practical deployment:
- Computational Bloat: State-of-the-art models are massive, requiring expensive GPUs and significant memory, making them difficult to run on CPUs or mobile devices.
- Context Blindness: Most models translate sentence-by-sentence, losing the "thread" of a story or document, which leads to inconsistent pronouns and disjointed narratives.
- The "One Best Answer" Fallacy: Standard MT aims for a single "correct" output. However, in fields like language education (e.g., Duolingo), a system needs to recognize all valid ways to translate a prompt to provide meaningful feedback to students.
Methodologies for Efficient and Contextual NMT
The proceedings highlight several innovative technical approaches:
- Knowledge Distillation: Using a massive "teacher" model to train a much smaller "student" model was a cornerstone for the Efficiency Task.
- Quantization: Researchers pushed the boundaries of precision, moving from 32-bit floats to 8-bit and even 4-bit logarithmic quantization. By leaving bias terms uncompressed, they achieved massive size reductions with negligible quality loss.
- Architectural Asymmetry: A recurring theme was using deep encoders (up to 35 layers) but very shallow decoders (often just 1 layer). Since the decoder is the bottleneck during inference, this shift provides a massive speedup.
- Consensus Voting and Filtering: For the STAPLE task, teams used "n-best" list generation followed by binary classifiers to filter out translations that, while technically correct, didn't sound like something a human learner would actually say.
Shared Tasks and Datasets
The workshop was structured around three primary shared tasks:
- Efficiency Task: Utilized the standard WMT 2019 English-German benchmarks to test speed and size.
- Document-level Generation and Translation (DGT): Used the Rotowire dataset, requiring systems to generate descriptive summaries from NBA basketball box-score tables.
- STAPLE Task: Introduced a novel dataset from Duolingo containing English prompts paired with comprehensive, weighted lists of acceptable human translations in five languages. Weights reflect "Learner Response Frequency" (LRF).
Key Insights and Results
The Decoder Bottleneck and Quantization
In the Efficiency Task, the "aha moment" was realizing that the decoder's autoregressive nature is where the time is lost. Pruning the decoder while keeping the encoder deep is the most effective way to save time without losing understanding. Furthermore, NMT model weights are skewed toward zero; logarithmic quantization captures this distribution much better than standard fixed-point methods, allowing models to shrink by 8x while staying within 0.2 BLEU points of the original.
BLEU vs. Diversity
A major finding in the STAPLE task was that models with the highest BLEU scores often performed poorly on the task's actual metric (Weighted F1). This suggests that training for the "average" translation actually hurts a model's ability to generate the "long tail" of valid paraphrases.
Performance Benchmarks
- Speed: The fastest systems could translate 1 million lines in under 3 seconds on a single GPU. On CPU, the OpenNMT team achieved a 13x speedup over their baseline with only a 0.8 BLEU point drop.
- Size: Models were successfully compressed from ~250MB down to under 20MB using 4-bit quantization.
- STAPLE Performance: The top-performing team (JHU) swept all five language tracks, outperforming the Amazon Translate baseline by factors of 2 to 10 in weighted F1 score.
- Document Context: The FJWU team showed that hierarchical attention (looking at previous sentences) improved translation quality by over 2 BLEU points compared to sentence-level baselines.
Limitations and Future Work
Despite the progress, several challenges remain. While 4-bit quantization saves disk space, current hardware doesn't natively support 4-bit math instructions, meaning speed gains aren't yet as large as size gains. Systems still struggle with highly inflected languages like Hungarian, and the DGT task had low participation, highlighting the difficulty of finding document-level parallel data. Future directions include moving toward reinforcement learning with semantic rewards (like XSIM) rather than relying on simple word-overlap metrics like ROUGE or BLEU.
Summary updated