Inference Strategies for Machine Translation with Conditional Masking

Julia Kreutzer, George F. Foster, Colin Cherry

Abstract

Conditional masked language model (CMLM) training has proven successful for non-autoregressive and semi-autoregressive sequence generation tasks, such as machine translation. Given a trained CMLM, however, it is not clear what the best inference strategy is. We formulate masked inference as a factorization of conditional probabilities of partial sequences, show that this does not harm performance, and investigate a number of simple heuristics motivated by this perspective. We identify a thresholding strategy that has advantages over the standard “mask-predict” algorithm, and provide analyses of its behavior on machine translation tasks.

Introduction

Standard Neural Machine Translation (NMT) relies on autoregressive decoding, generating tokens sequentially from left to right. While accurate, this process is inherently slow. Conditional Masked Language Models (CMLMs) offer a faster alternative by iteratively refining a sequence of "MASK" tokens. However, the conventional "mask-predict" algorithm employs a fixed iteration schedule, which fails to account for varying sentence complexity. This leads to inefficiencies and "multimodal" errors, such as word repetitions. This paper explores optimized inference strategies that adapt the decoding pace based on model confidence.

Methodology

The authors reframe CMLM inference as a probabilistic factorization. They introduce the update-masked-sub strategy, where tokens are unmasked based on confidence and never re-masked, providing a mathematically consistent sequence of conditional probabilities. Central to this approach are new unmasking heuristics:

  • thresh: Unmasks all tokens exceeding a probability threshold $\tau$.
  • comb-thresh: Unmasks the largest set of tokens whose joint probability (the product of individual scores) exceeds $\tau$.
  • fcomb-thresh: A calibrated variation of comb-thresh that accounts for the probability of tokens not chosen.

These heuristics allow the model to dynamically adjust its speed, spending more iterations on complex linguistic structures and fewer on simple ones.

Results and Analysis

Evaluation on WMT14 English-German (en-de) and WMT17 Chinese-English (zh-en) benchmarks, using sequence-level knowledge distillation, demonstrates the superiority of thresholding strategies.

Quality vs speedup trade-off on WMT En-De.

As shown in the quality-speedup trade-off (Figure 3), the comb-thresh strategy consistently outperforms the standard mask-predict algorithm across different speed settings. On the English-German task, it achieves a theoretical 4x speedup over a beam-search autoregressive Transformer with a negligible loss of less than 0.3 BLEU points.

Number of iterations vs sentence length for different heuristics.

A key insight is the adaptive nature of these heuristics. Unlike fixed-iteration methods, thresholding naturally scales the number of iterations with sentence length (Figure 2). This "confidence-based pacing" ensures that the model provides sufficient context to resolve ambiguities in longer, more complex sentences, effectively acting as a "mode-breaker" to prevent common parallel generation errors like stuttering.

Conclusion

Moving away from rigid decoding schedules to confidence-based thresholding significantly improves the efficiency and quality of CMLM inference. While requiring some hyperparameter tuning for the threshold $\tau$, these strategies offer a more intuitive and effective framework for non-autoregressive machine translation. Future work will explore their applicability to other sequence generation tasks like summarization.

Summary updated

Abstract

Conditional masked language model (CMLM) training has proven successful for non-autoregressive and semi-autoregressive sequence generation tasks, such as machine translation. Given a trained CMLM, however, it is not clear what the best inference strategy is. We formulate masked inference as a factorization of conditional probabilities of partial sequences, show that this does not harm performance, and investigate a number of simple heuristics motivated by this perspective. We identify a thresholding strategy that has advantages over the standard “mask-predict” algorithm, and provide analyses of its behavior on machine translation tasks.
BibTeX
@inproceedings{kreutzer2020inference,
  author = {Julia Kreutzer and George F. Foster and Colin Cherry},
  title = {{Inference Strategies for Machine Translation with Conditional Masking}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=JWn_36Ygnfl},
}
Figure