Abstract
This paper introduces DCMCL (Diverse Context Modelling with Collaborative Learning), a framework that treats Autoregressive (AR) and Non-autoregressive (NAR) models as equal collaborators rather than teacher and student. By combining token-level mutual learning with sequence-level contrastive learning, the authors allow both models to benefit from each other's unique contextual strengths, leading to significant performance gains in translation quality for both architectures across multiple standard benchmarks.
Introduction
In Neural Machine Translation (NMT), two primary paradigms exist: Autoregressive (AR) models and Non-autoregressive (NAR) models. AR models predict tokens sequentially, capturing local dependencies effectively but suffering from slow inference speeds and unidirectional context. Conversely, NAR models predict all tokens simultaneously, offering high speed but often falling victim to the "multi-modality problem," where the lack of token dependency leads to repetitions or omissions.
Traditionally, these models are linked via Knowledge Distillation, where a static AR "teacher" trains an NAR "student." This paper argues that this one-way relationship is suboptimal. It proposes a collaborative learning approach where both models evolve together, sharing their diverse perspectives—AR's local coherence and NAR's bidirectional global context—to improve overall translation quality.
The DCMCL Framework
The proposed DCMCL framework utilizes a shared encoder and two distinct decoders (AR and NAR). The core innovation lies in its hierarchical learning strategy designed to align and improve both models simultaneously.
Hierarchical Learning Strategy
- Token-level Mutual Learning (TML): The framework minimizes the Kullback-Leibler (KL) divergence between the probability distributions predicted by both decoders. This encourages the AR model to incorporate bidirectional context from the NAR model, while the NAR model learns from the AR's superior local flow.
- Sequence-level Contrastive Learning (SCL): To ensure semantic alignment, SCL treats the outputs of the AR and NAR decoders for the same source sentence as a "positive pair." By pushing these representations closer in the semantic space and away from other samples in the batch, the models achieve a unified high-level understanding of the translation task.
- Hybrid Teacher (DCMCL-HYB): An advanced variant fuses features from both models to create a "union context," providing a more stable and informative anchor for training than either model could provide in isolation.
Experimental Results
The researchers evaluated DCMCL on several standard NMT benchmarks, including WMT14 (EN-DE, DE-EN), WMT16 (EN-RO, RO-EN), and IWSLT datasets. The framework was tested on both raw and distilled data to ensure robustness.
Key Findings
- Significant BLEU Gains: DCMCL consistently improved performance, with AR models seeing gains of up to 1.38 BLEU points and NAR models improving by up to 2.98 BLEU points on IWSLT datasets. As shown in the results table, DCMCL-enhanced models (lines 19-20) frequently outperformed state-of-the-art unified models like JANUS and Diformer.
- Mitigating Multi-modality: Collaborative training significantly reduced the percentage of repeated tokens in NAR outputs, indicating a better grasp of sentence structure.
- Scalability: The method proved effective even when scaled to the massive WMT21 dataset (over 100 million sentences), demonstrating its viability for large-scale production environments.
Conclusion
DCMCL shifts the paradigm from one-way knowledge distillation to mutual collaboration. By leveraging the complementary strengths of AR and NAR contexts, the framework achieves superior translation quality for both model types. While challenges remain regarding positional embedding sensitivity and increased training complexity, the success of DCMCL suggests that collaborative learning is a powerful tool for generative tasks where diverse modeling perspectives are available.
Summary updated