Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation

Weijia Xu, Xing Niu, Marine Carpuat

Abstract

Semi-supervised learning has become a cornerstone of Neural Machine Translation (NMT), leveraging vast amounts of monolingual data to supplement limited parallel corpora. Two prominent methods, Iterative Back-Translation (IBT) and Dual Learning, have emerged as effective but heuristically distinct approaches. This paper introduces the Dual Reconstruction Objective, a unified theoretical framework that reveals IBT and Dual Learning as different approximations of the same underlying goal: maximizing the marginal likelihood of monolingual data.

Introduction

In NMT, the scarcity of human-translated parallel data is often the primary bottleneck. To address this, researchers utilize monolingual text through techniques like IBT, which uses a "round-trip" translation approach, and Dual Learning, which employs reinforcement learning and auxiliary language models. Despite their success, the theoretical relationship between these methods remained opaque. This work bridges that gap, demonstrating that a simpler approach—IBT—often approximates the ideal training objective more effectively than its more complex counterparts.

The Dual Reconstruction Framework

The authors propose the Dual Reconstruction Objective ($J_{dual}$) as a unifying principle. This objective treats NMT as a bidirectional process where models are trained to reconstruct original sentences after a round-trip translation through a latent "pseudo-translation."

As illustrated in the framework diagram below, the process involves two paths:

  1. Target-Source-Target: A target sentence $y$ is translated into a pseudo-source $\tilde{x}$ by an inference model $q_\phi$, and then reconstructed back to $y$ by the NMT model $p_\theta$.
  2. Source-Target-Source: A source sentence $x$ is translated into a pseudo-target $\tilde{y}$ and reconstructed back to $x$.
The Dual Reconstruction framework showing the two-way translation and reconstruction paths.

Mathematically, the paper proves that optimizing this dual reconstruction objective is equivalent to maximizing the marginal likelihood of monolingual data, provided certain mutual information constraints are met. Crucially, the analysis shows that IBT is a specific approximation of this objective where gradients are "blocked" (treating pseudo-translations as fixed), whereas Dual Learning attempts to compute these gradients using policy gradients.

Empirical Evaluation

The researchers conducted a rigorous comparison across three distinct NMT scenarios:

  • High-Resource: WMT18 German-English (4.5M parallel sentences).
  • Low-Resource: WMT18 Turkish-English (0.2M parallel sentences).
  • Cross-Domain: Training on WMT News and testing on IWSLT17 TED talks.

Key Results and Insights

The experiments yielded several surprising insights that challenge the necessity of complex semi-supervised architectures:

  • IBT Outperforms Dual Learning: In head-to-head comparisons, epoch-level IBT consistently achieved higher BLEU scores. For instance, in the low-resource Turkish-English task, IBT reached a BLEU score of 19.38, significantly higher than the 17.85 achieved by Dual Learning.
  • Redundancy of Language Models: The theoretical analysis suggests that the extra language model loss used in Dual Learning is redundant; the reconstruction objective itself naturally guides the model toward the correct language distribution.
  • Efficiency Gains: Batch-level IBT (translating on-the-fly) was found to be 30–50% faster to train than Dual Learning while maintaining comparable or superior translation quality.
Dual reconstruction loss across low-resource, high-resource, and cross-domain tasks, comparing batch-level IBT with various dual learning configurations.

As shown in the loss curves above, batch-level IBT (blue squares) consistently achieves lower or competitive reconstruction loss compared to various configurations of dual learning, particularly in high-resource and cross-domain settings.

Discussion and Future Work

The "less is more" result of this study highlights that complexity does not always equate to better performance in semi-supervised NMT. By providing a unified objective, the authors have simplified the landscape of NMT training, showing that IBT is not just a heuristic but a theoretically sound approximation of the marginal likelihood.

Future work may explore applying the dual reconstruction objective to other sequence-to-sequence tasks, such as abstractive summarization or dialogue generation. Additionally, while IBT is effective, the computational cost of "epoch-level" translation remains high, leaving room for research into more efficient high-quality approximation methods.

Summary updated

BibTeX
@inproceedings{xu2020dual,
  author = {Weijia Xu and Xing Niu and Marine Carpuat},
  title = {{Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=QhnXRUdhkEY},
}
Figure