Non-autoregressive Machine Translation with Disentangled Context Transformer

Jungo Kasai, James Cross, Marjan Ghazvininejad, Jiatao GuICML 2020 · OpenReview

This paper, presented at ICML 2020 by researchers at the University of Washington and Facebook AI, introduces the DisCo (Disentangled Context) Transformer. This novel approach to non-autoregressive machine translation bridges the gap between the speed of parallel generation and the accuracy of traditional sequential models.

Overview

The DisCo Transformer enables fast, non-autoregressive machine translation by predicting all words in a sentence simultaneously, each conditioned on a different random subset of the other words. Combined with a novel "parallel easy-first" inference algorithm, it achieves state-of-the-art non-autoregressive performance with significantly fewer decoding iterations than previous methods.

The Challenge of Non-Autoregressive Translation

Standard Neural Machine Translation (NMT) models are autoregressive: they generate text one word at a time, from left to right. While accurate, this is slow because the time complexity scales linearly with the sentence length.

Non-autoregressive translation (NAT) aims to generate all words in parallel. However, NAT faces two major hurdles:

  1. The Multimodality Problem: Because words are generated independently, the model might pick pieces from different valid translations that don't fit together (e.g., "Thank you" vs. "I thank you" resulting in "I thank you you").
  2. Training Inefficiency: Previous iterative NAT models, like Conditional Masked Language Models (CMLM), only learn to predict "masked" (hidden) words during training. This makes training slow and prevents the model from learning from the entire context of the sentence in a single pass.

Methodology: DisCo Transformer

The authors propose two primary innovations to overcome these hurdles:

1. The DisCo Objective and Architecture

Unlike CMLMs that only predict a subset of masked tokens, the DisCo objective trains the model to predict every token in a sentence, each conditioned on a different, randomly sampled subset of the other tokens.

Architecture of the DisCo Transformer showing attention masking and decontextualized Keys/Values.

To make this computationally feasible, they use a specific attention masking strategy. In a single forward pass, the model can compute predictions for all words, each seeing a different "disentangled" context. To prevent "information leakage" (where a word accidentally sees itself in a higher layer of the network), they decontextualize the Keys and Values in the Transformer layers, making them depend only on the input word and its position, rather than the previous layer's hidden states.

2. Parallel Easy-First Inference

The authors introduce a new decoding algorithm. Instead of a fixed number of refinement steps, they use an "easy-first" approach:

  • Iteration 1: Predict all words in parallel based only on the source text.
  • Subsequent Iterations: Rank the words by the model's confidence. In the next step, the model re-predicts every word, but this time it is allowed to "see" the words it was most confident about in the previous step.
  • Termination: The process repeats until the predictions converge, allowing the model to adaptively decide how many steps it needs for a specific sentence.

Evaluation Datasets

The model was evaluated on seven translation directions across four standard benchmarks:

  • WMT14 EN-DE (4.5M pairs)
  • WMT16 EN-RO (610K pairs)
  • WMT17 EN-ZH (20M pairs)
  • WMT14 EN-FR (36M pairs)

The variety in dataset sizes allowed the authors to demonstrate how the model scales with more data.

Key Insights

  • Efficiency is King: By predicting all tokens at once during training, DisCo is much more sample-efficient than masked language models. It essentially gets $N$ times more "supervision" per sentence (where $N$ is sentence length).
  • Confidence as a Guide: The "easy-first" strategy mimics how humans might translate a difficult sentence—filling in the certain parts first to provide context for the ambiguous parts.
  • Generalization: The authors show that the DisCo objective is a mathematical generalization of other popular objectives, including BERT (cloze) and XLNet (permutation-based).
  • Robustness to Batch Size: DisCo proved more robust to smaller training batch sizes than CMLM, likely due to its more efficient use of the available context in every training step.

Results and Performance

  • Performance: DisCo achieved competitive (and sometimes superior) BLEU scores compared to state-of-the-art NAT models like the Levenshtein Transformer and CMLM.
  • Speedup: On the EN-ZH task, DisCo achieved a 1.4 BLEU gain over the CMLM baseline while requiring only 5.44 iterations on average, compared to the 10 iterations typically used by CMLM.
Relative speedup versus BLEU score on the EN-ZH translation task.
  • Scaling: The performance gains were most pronounced on larger datasets (EN-ZH and EN-FR), where DisCo's training efficiency allowed it to approach the quality of traditional autoregressive models.
  • Wall-clock Speed: The model provides a roughly 3x to 5x speedup over standard autoregressive Transformers during inference.

Limitations and Future Directions

  • The AR Gap: While DisCo closes the gap significantly, a small performance difference still exists between NAT and the best autoregressive "teacher" models, especially on complex, long-form text.
  • Dependence on Distillation: Like almost all NAT models, DisCo relies on Knowledge Distillation (training on the output of an autoregressive model) to simplify the data distribution.
  • Future Directions: The authors suggest that DisCo's architecture could be used for general-purpose representation learning (pre-training), similar to BERT, but with more flexibility in how context is handled.

Summary updated

Abstract

State-of-the-art neural machine translation models generate a translation from left to right and every step is conditioned on the previously generated tokens. The sequential nature of this generati...
BibTeX
@inproceedings{kasai2020non,
  author = {Jungo Kasai and James Cross and Marjan Ghazvininejad and Jiatao Gu},
  title = {{Non-autoregressive Machine Translation with Disentangled Context Transformer}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=ZbVlWiY4sKi},
}
Figure