Bi-Directional Differentiable Input Reconstruction for Low-Resource Neural Machine Translation

Xing Niu, Weijia Xu, Marine Carpuat

Abstract

We aim to better exploit the limited amounts of parallel text available in low-resource settings by introducing a differentiable reconstruction loss for neural machine translation (NMT). This loss compares original inputs to reconstructed inputs, obtained by back-translating translation hypotheses into the input language. We leverage differentiable sampling and bi-directional NMT to train models end-to-end, without introducing additional parameters. This approach achieves small but consistent BLEU improvements on four language pairs in both translation directions, and outperforms an alternative differentiable reconstruction strategy based on hidden states.

Introduction

Neural Machine Translation (NMT) typically requires vast amounts of parallel data to achieve high performance. In low-resource settings, performance degrades significantly. To address this, researchers have explored "round-trip translation"—reconstructing the source from the target hypothesis—to provide additional training signals. However, standard discrete word selection makes this process non-differentiable, often requiring complex and unstable Reinforcement Learning. This paper introduces a differentiable reconstruction loss using Gumbel-Softmax sampling and bi-directional parameter sharing to enable stable, end-to-end training.

Methodology

The proposed approach focuses on making input reconstruction both compact and differentiable:

  1. Differentiable Sampling: By employing the Straight-Through Gumbel Softmax (STGS) estimator, the model can sample intermediate words while maintaining a gradient path for backpropagation. This allows the reconstruction error to directly inform the original translation step.
  2. Bi-Directional Parameter Sharing: A single NMT model is trained for both directions (e.g., EN ↔ SW) using language tags. This parameter efficiency is vital for low-resource scenarios where data is insufficient to support large, separate models.

Experimental Setup

The method was evaluated on four low-resource language pairs: Swahili (SW), Tagalog (TL), Somali (SO), and Turkish (TR), all paired with English (EN). Dataset sizes ranged from approximately 60k to 207k parallel sentences. Models were pre-trained on standard translation tasks before enabling the reconstruction loss to ensure a stable training signal.

Results and Analysis

The sampling-based reconstruction method consistently outperformed a baseline that reconstructs from hidden states. As shown in Figure 2, the "Hidden" approach quickly achieves low reconstruction perplexity on the training set but fails to improve translation perplexity on the development set. This suggests the model merely learns to copy input information into hidden states (the "Hidden State Trap") rather than improving translation quality.

Perplexity comparison between Hidden state and Sampling-based reconstruction showing the training dynamics.

In contrast, the proposed sampling method leads to better generalization. Quantitative results in the table below demonstrate small but consistent BLEU score improvements across all eight translation directions, with gains typically between +0.2 and +0.7 BLEU over the baseline.

BLEU score improvements across four language pairs for different model configurations.

Discussion and Conclusion

While the gains are modest, they are achieved without adding any new parameters, making the technique highly suitable for resource-constrained environments. The stability of the Gumbel-Softmax approach over hidden-state reconstruction is a key finding. Future work may extend this technique to monolingual data via back-translation, potentially unlocking even greater performance gains by leveraging larger unlabelled corpora.

Summary updated

Abstract

We aim to better exploit the limited amounts of parallel text available in low-resource settings by introducing a differentiable reconstruction loss for neural machine translation (NMT). This loss compares original inputs to reconstructed inputs, obtained by back-translating translation hypotheses into the input language. We leverage differentiable sampling and bi-directional NMT to train models end-to-end, without introducing additional parameters. This approach achieves small but consistent BLEU improvements on four language pairs in both translation directions, and outperforms an alternative differentiable reconstruction strategy based on hidden states.
BibTeX
@inproceedings{niu2019bi,
  author = {Xing Niu and Weijia Xu and Marine Carpuat},
  title = {{Bi-Directional Differentiable Input Reconstruction for Low-Resource Neural Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=HkVjrXW_Zr},
}
Figure