A Correlational Encoder Decoder Architecture for Pivot Based Sequence Generation

Amrita Saha, Mitesh M. Khapra, Sarath Chandar, Janarthanan Rajendran, Kyunghyun Cho

Abstract

In many sequence-to-sequence generation tasks, such as machine translation and transliteration, parallel data between a specific source domain ($X$) and target domain ($Y$) is often unavailable. This paper, presented at COLING 2016, proposes a novel Correlational Encoder-Decoder architecture that leverages a third "pivot" domain ($Z$) for which parallel data exists for both $(X, Z)$ and $(Z, Y)$. By employing a correlation-based loss function, the model learns a shared representation space, enabling direct $X \to Y$ generation during inference without the need for intermediate pivot generation.

Introduction

The efficacy of modern neural machine translation (NMT) relies heavily on the availability of large-scale parallel corpora. However, as the number of languages ($n$) increases, the number of potential language pairs grows quadratically ($nC2$), making it impossible to acquire parallel data for every combination. Traditional approaches address this "zero-resource" problem using a two-stage pipeline: first translating $X$ to a pivot language $Z$ (typically English), and then translating $Z$ to the target $Y$.

This pipeline approach suffers from two primary drawbacks: error propagation, where inaccuracies in the first stage compound in the second, and computational inefficiency, requiring the execution of two complete encoder-decoder models. This research introduces a single-model architecture inspired by the concept of a "Vectorial Interlingua"—a shared mathematical coordinate system where different languages or modalities are mapped into the same semantic space.

Methodology: The Correlational Encoder-Decoder

The proposed architecture consists of two encoders (for source $X$ and pivot $Z$) and one decoder (for target $Y$). The training strategy is designed to make the representations of $X$ and $Z$ interchangeable.

Joint Training and Correlation Loss

The model is trained using two simultaneous objectives:

  1. Correlation Loss: Using the $(X, Z)$ dataset, the model forces the encoders to produce hidden representations that are highly correlated. A standardization function ensures these vectors have zero mean and unit variance, facilitating a shared latent space.
  2. Cross-Entropy Loss: Using the $(Z, Y)$ dataset, the model trains the decoder to generate the target $Y$ from the pivot's representation.

By alternating between these tasks, the model learns that the representation for $X$ should occupy the same space as the representation for $Z$. At test time, the pivot $Z$ is discarded; the model takes $X$, passes it through the $X$-encoder, and feeds the resulting vector directly into the $Y$-decoder.

The Correlational Encoder-Decoder architecture aligns source and pivot representations to enable direct target generation.

Experimental Evaluation

The researchers evaluated the model on two distinct tasks:

  1. Bridge Transliteration: Using the NEWS 2012 dataset, the authors tested 12 zero-resource language pairs among Hindi (Hi), Kannada (Ka), Tamil (Ta), and Marathi (Ma), using English as the pivot.
  2. Bridge Captioning: Using MSCOCO, the model attempted to generate French captions from images. The training used Image-English pairs and a pseudo-parallel English-French corpus generated via machine translation.

Results and Analysis

In the transliteration task, the Correlational Encoder-Decoder demonstrated significant improvements over traditional methods. It outperformed a strong Two-Stage Phrase-Based Statistical Machine Translation (PBSMT) baseline in 11 out of 12 language pairs. Furthermore, it surpassed a two-stage neural baseline in 50% of the cases, validating that joint training can mitigate the error propagation inherent in pipelined systems.

Quantitative results showing the Correlational Encoder-Decoder outperforming two-stage baselines in transliteration tasks.

In the more complex bridge captioning task, the model successfully generated relevant French descriptions for images (e.g., identifying "un sandwich" or "un bus"). While it did not beat the two-stage baseline in this specific modality—likely due to the high quality of the specialized translation engine used in the baseline's second stage—it proved the viability of the correlational approach for multimodal tasks.

Conclusion and Future Directions

The Correlational Encoder-Decoder offers a resource-efficient path for zero-resource sequence generation. By treating the pivot language as a scaffold during training, the model learns a mathematical interlingua that bridges domains without $n^2$ datasets. Future work aims to incorporate Attention Mechanisms to handle longer, more complex sequences and to address the domain shift challenges present in multimodal captioning.

Summary updated

BibTeX
@inproceedings{saha2016correlational,
  author = {Amrita Saha and Mitesh M. Khapra and Sarath Chandar and Janarthanan Rajendran and Kyunghyun Cho},
  title = {{A Correlational Encoder Decoder Architecture for Pivot Based Sequence Generation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
  url = {https://openreview.net/forum?id=BZENlCVSp19},
}
Figure