Bi-Decoder Augmented Network for Neural Machine Translation

Boyuan Pan, Yazheng Yang, Zhou Zhao, Yueting Zhuang, Deng Cai

Abstract

Neural Machine Translation (NMT) typically relies on unidirectional encoder-decoder frameworks that can lead to language-pair-specific encoders. This paper introduces the Bi-Decoder Augmented Network (BiDAN), which utilizes an auxiliary decoder to reconstruct the source sentence during training. By forcing a shared encoder to satisfy both translation and reconstruction tasks, BiDAN encourages the learning of more language-independent semantic representations. Incorporating denoising and reinforcement learning strategies ensures the auxiliary task captures deep semantics rather than trivial sequences. Experimental results across multiple benchmarks, including WMT14 and IWSLT15, demonstrate significant improvements in translation quality, with gains of up to 2.2 BLEU points over baseline models.

Introduction

In standard NMT, the encoder-decoder framework maps a source language sentence directly to a target language sentence. While effective, this approach often results in an encoder that is overly specialized to the specific language pair it was trained on. A truly robust NMT system should ideally summarize text into a language-independent semantic space—an "interlingua"—before generating the target text. While existing research has focused on shared word embeddings, BiDAN tackles the challenge of creating language-independent representations at higher semantic levels where complex phrases and sentence structures are processed.

Methodology

The core innovation of BiDAN is the dual-decoder architecture. Instead of a single decoder, the model employs two decoders sharing a single encoder $\mathcal{E}$:

  1. Original Decoder ($\mathcal{D}_1$): Generates the target language sequence (e.g., English to German).
  2. Auxiliary Decoder ($\mathcal{D}_2$): Reconstructs the original source sentence from the encoder's hidden states.
The BiDAN architecture featuring a shared encoder and dual decoders for translation and reconstruction.

To ensure the auxiliary decoder helps the encoder learn deep semantics rather than performing a trivial "copy-paste" task, the authors employ three specific strategies:

  • Denoising: During training, the word order of the source sentence is randomly shuffled. The auxiliary decoder must reconstruct the correct order, forcing the encoder to capture internal logic and meaning.
  • Reinforcement Learning (RL): A reward function based on cosine similarity between the embeddings of predicted and actual source text is used. This encourages the model to focus on the "gist" of the sentence, allowing for synonyms rather than penalizing non-exact word matches.
  • Multi-Objective Learning: Training is staged, starting with the main translation task, then alternating between translation, denoising, and RL-based reconstruction, and finally fine-tuning the main translation task.

Experimental Results

The researchers evaluated BiDAN on three NMT benchmarks: WMT14 English-to-German (En→De), WMT14 English-to-French (En→Fr), and IWSLT 2015 English-to-Vietnamese (En→Vi). BiDAN consistently outperformed baseline models across all datasets.

Main results and ablation study showing the impact of the auxiliary decoder (AD), denoising, and RL components across different benchmarks.

As shown in the ablation studies, simply adding the auxiliary decoder (AD) provided the most significant initial performance jump. The addition of Denoising and RL components provided the final refinements to reach peak BLEU scores. Specifically, BiDAN improved BLEU scores by +2.1 for En→De, +1.9 for En→Fr, and +2.2 for the low-resource En→Vi task.

Key Insights and Analysis

  • High-Level Independence: By applying the bi-decoder approach to the final layers of the encoder, the model achieves language independence at the "thought" level. This was evidenced by "zero-shot" encoder swap experiments, where a BiDAN-trained encoder performed significantly better in a new language pair than a standard encoder.
  • Architecture Compatibility: While primary experiments used LSTMs, the authors demonstrated that the Bi-Decoder concept also improves the performance of the state-of-the-art Transformer architecture.
  • The Trivial Copy Trap: The researchers found that without the "noise" introduced via denoising, the auxiliary decoder would not force the encoder to learn new, robust representations.

Conclusion

BiDAN offers a robust method for improving NMT by encouraging language-independent semantic representations through an auxiliary reconstruction task. While it increases computational overhead during training, the auxiliary decoder is discarded during inference, maintaining standard translation speeds. Future work may extend this multi-target approach to other modalities, such as joint translation and image generation, to further push the boundaries of language-independent representation.

Summary updated

BibTeX
@article{pan2020bi,
  author = {Boyuan Pan and Yazheng Yang and Zhou Zhao and Yueting Zhuang and Deng Cai},
  title = {{Bi-Decoder Augmented Network for Neural Machine Translation}},
  journal = {Neurocomputing},
  year = {2020},
  doi = {10.1016/j.neucom.2020.01.003},
}
Figure