Bi-Directional Differentiable Input Reconstruction for Low-Resource Neural Machine Translation

Xing Niu, Weijia Xu, Marine Carpuat

Abstract: Xing Niu, Weijia Xu, Marine Carpuat. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

2019.

Introduction

Neural Machine Translation (NMT) has revolutionized the field but remains heavily dependent on large-scale parallel corpora. For low-resource language pairs, where data is scarce, NMT performance often degrades significantly. This paper explores input reconstruction as a training signal: the idea that a model should be able to translate a sentence into a target language and then back into the original source language accurately.

The Challenges of Input Reconstruction

While intuitive, "round-trip" translation faces two primary technical hurdles:

  1. Differentiability: Standard NMT involves discrete sampling of words, which is non-differentiable. This prevents gradients from the reconstruction loss from flowing back to the initial translation parameters.
  2. Model Complexity: Previous dual-learning frameworks often required separate "reconstructor" models, doubling the parameter count and increasing training instability.

Proposed Methodology

The authors introduce a compact, end-to-end approach to address these issues:

Bi-Directional Modeling

Instead of separate models, a single bi-directional NMT model is trained for both Source $\rightarrow$ Target and Target $\rightarrow$ Source directions. By using language tags (e.g., <en>), the model acts as its own reconstructor, sharing weights and improving parameter efficiency—a critical factor in low-resource settings.

Differentiable Sampling with STGS

To enable end-to-end training, the researchers employ the Straight-Through Gumbel Softmax (STGS) estimator. This allows the model to perform discrete word sampling during the forward pass while using a continuous approximation for the backward pass, making the entire reconstruction process differentiable.

Experimental Results

The method was evaluated on four low-resource pairs: Swahili (SW), Tagalog (TL), Somali (SO), and Turkish (TR), all paired with English (EN).

BLEU score improvements across diverse language pairs.

As shown in the results table, the proposed sampling-based reconstruction method achieved consistent BLEU score improvements ranging from +0.19 to +0.69 points over a strong bi-directional baseline. These gains were statistically significant ($p < 0.01$) across all eight translation directions, whereas alternative methods like hidden-state reconstruction often degraded performance in English-to-foreign directions.

Analysis: Words vs. Hidden States

A key finding of the paper is the superiority of reconstructing from actual words (sampling) rather than internal "hidden states."

Reconstruction and translation perplexity curves comparing Hidden vs. Sampling methods.

The authors argue that hidden-state reconstruction is "artificially easy." As illustrated in the perplexity curves, the "Hidden" method (red dashed line) achieves extremely low reconstruction perplexity almost immediately, suggesting the model is simply over-memorizing or copying input into its internal vectors. In contrast, the "Sampling" method (blue solid line) maintains a higher reconstruction perplexity but leads to better overall translation performance on the development set, indicating a more robust and meaningful training signal.

Conclusion

This work demonstrates that differentiable input reconstruction is a viable and efficient way to boost low-resource NMT. By leveraging bi-directional models and STGS, the authors achieve consistent performance gains without increasing model size. Future work will explore applying these techniques to unsupervised NMT and semi-supervised learning with monolingual data.

Summary updated

BibTeX
@inproceedings{niu2019bi,
  author = {Xing Niu and Weijia Xu and Marine Carpuat},
  title = {{Bi-Directional Differentiable Input Reconstruction for Low-Resource Neural Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=HkVjrXW_Zr},
}
Figure