Abstract: Xing Niu, Weijia Xu, Marine Carpuat. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
2019.
Introduction
Neural Machine Translation (NMT) has revolutionized the field but remains heavily dependent on large-scale parallel corpora. For low-resource language pairs, where data is scarce, NMT performance often degrades significantly. This paper explores input reconstruction as a training signal: the idea that a model should be able to translate a sentence into a target language and then back into the original source language accurately.
The Challenges of Input Reconstruction
While intuitive, "round-trip" translation faces two primary technical hurdles:
- Differentiability: Standard NMT involves discrete sampling of words, which is non-differentiable. This prevents gradients from the reconstruction loss from flowing back to the initial translation parameters.
- Model Complexity: Previous dual-learning frameworks often required separate "reconstructor" models, doubling the parameter count and increasing training instability.
Proposed Methodology
The authors introduce a compact, end-to-end approach to address these issues:
Bi-Directional Modeling
Instead of separate models, a single bi-directional NMT model is trained for both Source $\rightarrow$ Target and Target $\rightarrow$ Source directions. By using language tags (e.g., <en>), the model acts as its own reconstructor, sharing weights and improving parameter efficiency—a critical factor in low-resource settings.
Differentiable Sampling with STGS
To enable end-to-end training, the researchers employ the Straight-Through Gumbel Softmax (STGS) estimator. This allows the model to perform discrete word sampling during the forward pass while using a continuous approximation for the backward pass, making the entire reconstruction process differentiable.
Experimental Results
The method was evaluated on four low-resource pairs: Swahili (SW), Tagalog (TL), Somali (SO), and Turkish (TR), all paired with English (EN).
As shown in the results table, the proposed sampling-based reconstruction method achieved consistent BLEU score improvements ranging from +0.19 to +0.69 points over a strong bi-directional baseline. These gains were statistically significant ($p < 0.01$) across all eight translation directions, whereas alternative methods like hidden-state reconstruction often degraded performance in English-to-foreign directions.
Analysis: Words vs. Hidden States
A key finding of the paper is the superiority of reconstructing from actual words (sampling) rather than internal "hidden states."
The authors argue that hidden-state reconstruction is "artificially easy." As illustrated in the perplexity curves, the "Hidden" method (red dashed line) achieves extremely low reconstruction perplexity almost immediately, suggesting the model is simply over-memorizing or copying input into its internal vectors. In contrast, the "Sampling" method (blue solid line) maintains a higher reconstruction perplexity but leads to better overall translation performance on the development set, indicating a more robust and meaningful training signal.
Conclusion
This work demonstrates that differentiable input reconstruction is a viable and efficient way to boost low-resource NMT. By leveraging bi-directional models and STGS, the authors achieve consistent performance gains without increasing model size. Future work will explore applying these techniques to unsupervised NMT and semi-supervised learning with monolingual data.
Summary updated