Abstract
Conditional masked language models (CMLM) have shown impressive progress in non-autoregressive machine translation (NAT). They learn the conditional translation model by predicting the random masked subset in the target sentence. Based on the CMLM framework, we introduce Multi-view Subset Regularization (MvSR), a novel regularization method to improve the performance of the NAT model. Specifically, MvSR consists of two parts: (1) shared mask consistency: we forward the same target with different mask strategies, and encourage the predictions of shared mask positions to be consistent with each other. (2) model consistency, we maintain an exponential moving average of the model weights, and enforce the predictions to be consistent between the average model and the online model. Without changing the CMLM-based architecture, our approach achieves remarkable performance on three public benchmarks with 0.36-1.14 BLEU gains over previous NAT models. Moreover, compared with the stronger Transformer baseline, we reduce the gap to 0.01-0.44 BLEU scores on small datasets (WMT16 RO↔EN and IWSLT DE→EN).
Introduction
Neural Machine Translation (NMT) has seen a paradigm shift with the advent of the Transformer architecture. However, the standard Transformer is inherently autoregressive, generating tokens sequentially from left to right. This sequential dependency creates a significant latency bottleneck, particularly for long sequences or real-time applications. Non-Autoregressive Translation (NAT) was proposed to overcome this by predicting all target tokens simultaneously.
While NAT offers substantial speedups, it historically lags behind autoregressive models in translation quality. This performance gap is primarily attributed to the "multimodality problem": because the model predicts tokens independently, it struggles to capture the complex dependencies between words in a sentence. Conditional Masked Language Models (CMLM) have mitigated this by using iterative refinement, but a significant quality gap remains. MvSR-NAT addresses this by introducing a dual-consistency regularization framework that stabilizes training and enhances the model's robustness.
Methodology: Multi-view Subset Regularization
The core of the proposed approach is Multi-view Subset Regularization (MvSR), which enhances the CMLM framework through two primary components: Shared Mask Consistency and Model Consistency.
Shared Mask Consistency
In a standard CMLM, the model is trained to predict a subset of masked tokens given the unmasked tokens and the source sentence. MvSR extends this by generating two different "views" of the same target sentence using distinct random masking patterns. The model is then trained to ensure that the probability distributions for tokens masked in both views (the shared masks) are consistent.
This approach treats shared masked positions as a "positive pair." By forcing consistency across different contexts (different sets of unmasked tokens), the model learns to be invariant to the specific masking noise and develops a more robust understanding of the underlying semantics. This is particularly beneficial for bridging the gap between training—where many tokens are unmasked—and inference, where the model often starts with a fully masked sequence.
Model Consistency
Complementing the data-level consistency is a model-level regularization inspired by the "Mean Teacher" paradigm. The researchers maintain an Exponential Moving Average (EMA) of the model's weights, referred to as the "average model." During training, the "online model" (the one being updated via backpropagation) is penalized for discrepancies between its predictions and those of the average model.
As shown in Figure 5, this dual-consistency approach ensures that the model remains stable despite the stochastic nature of dropout and random masking. Crucially, these regularization terms are only applied during training; the inference architecture remains a standard CMLM, ensuring zero additional computational overhead for the end-user.
Experiments and Results
The MvSR-NAT model was evaluated on three standard benchmarks: WMT14 English-German (EN-DE), WMT16 English-Romanian (EN-RO), and IWSLT14 German-English (DE-EN). To simplify the target distribution, Knowledge Distillation (KD) was employed, using a pre-trained autoregressive Transformer as the teacher model.
Performance Gains
MvSR-NAT consistently outperformed the baseline CMLM-NAT across all datasets, with BLEU score improvements ranging from 0.36 to 1.14 points. On smaller datasets, the performance gap with the state-of-the-art autoregressive Transformer was nearly eliminated. For instance, on WMT16 RO-EN, the gap was reduced to a mere 0.01 BLEU points.
Training Stability
The inclusion of MvSR significantly improves training stability. Figure 6 displays the BLEU score progression over 100 epochs.
The curves demonstrate that while the vanilla CMLM baseline (blue line) plateaus early, the MvSR-NAT model (red line) continues to improve, ultimately reaching a higher peak. The ablation studies confirm that both shared mask consistency and model consistency contribute to the overall performance boost, with shared mask consistency generally providing a more substantial individual gain.
Inference Speed
Despite the significant quality improvements, MvSR-NAT retains the speed advantages of non-autoregressive models. Depending on the number of refinement iterations used during decoding, the model achieves speedups of 3.77x to 9.79x compared to the autoregressive Transformer baseline.
Conclusion
MvSR-NAT introduces a powerful yet simple regularization strategy for non-autoregressive translation. By enforcing consistency across multiple views of both the data and the model parameters, MvSR-NAT effectively addresses the training-inference gap and the multimodality problem inherent in NAT. The results demonstrate that it is possible to achieve translation quality comparable to autoregressive models while maintaining the high-speed parallel generation that makes NAT so promising for real-world applications.
Summary updated