A Correlational Encoder Decoder Architecture for Pivot Based Sequence Generation

Amrita Saha, Mitesh M. Khapra, Sarath Chandar, Janarthanan Rajendran, Kyunghyun Cho

Abstract

Interlingua-based Machine Translation (MT) aims to encode multiple languages into a common linguistic representation and then decode sentences in multiple target languages from this representation. In this work, we explore this idea in the context of neural encoder-decoder architectures, albeit on a smaller scale and without MT as the end goal. Specifically, we consider the case of three languages or modalities $X$, $Z$, and $Y$ wherein we are interested in generating sequences in $Y$ starting from information available in $X$. However, there is no parallel training data available between $X$ and $Y$, but training data is available between $X$ & $Z$ and $Z$ & $Y$. $Z$ thus acts as a pivot/bridge. Instead of a two-stage model, we explore an interlingua-inspired solution which jointly learns to encode $X$ and $Z$ to a common representation and decode $Y$ from this common representation. We evaluate our model on bridge transliteration and bridge captioning, reporting promising results.

Introduction

In many real-world sequence generation tasks, direct parallel training data between a source domain ($X$) and a target domain ($Y$) is unavailable. For instance, translating between two low-resource languages or generating captions in a specific language for images often lacks direct supervision. However, data is frequently available between these domains and a common "pivot" domain ($Z$), such as English.

The standard workaround is a two-stage system: translate from $X$ to $Z$, then translate that output into $Y$. While functional, this "cascading" approach is prone to error propagation from the first stage and fails to learn a truly unified representation of the underlying meaning. This paper proposes a Correlational Encoder-Decoder (CED) architecture that moves toward an interlingua—a common linguistic representation where any language can be encoded into a shared space and decoded into any other.

The Correlational Encoder-Decoder Architecture

The proposed model aims to jointly learn representations for the source and pivot domains while training a decoder for the target domain.

The Correlational Encoder-Decoder architecture for joint learning across source (X), pivot (Z), and target (Y) domains.

The architecture consists of two encoders (for $X$ and $Z$) and one decoder (for $Y$). The training process involves two primary objectives:

  1. Correlation Maximization: A correlation loss forces the encoders for $X$ and $Z$ to produce highly similar latent vectors for parallel $(X, Z)$ pairs. This ensures that "cat" in $X$ and "cat" in $Z$ map to the same region in the shared latent space.
  2. Cross-Entropy Minimization: The decoder is trained to generate the correct sequence in $Y$ using the representation provided by the $Z$ encoder from parallel $(Z, Y)$ pairs.

Because the $X$ and $Z$ representations are forced to be similar, at inference time, the $X$ encoder can be plugged directly into the $Y$ decoder. The model generates $Y$ from $X$ without ever actually "stepping through" the pivot language $Z$.

Experimental Evaluation

The researchers evaluated the CED model on two distinct tasks: bridge transliteration and bridge captioning.

Bridge Transliteration

Using the NEWS 2012 dataset, the model performed transliteration between four Indian languages (Hindi, Kannada, Tamil, and Marathi) using English as the pivot. The CED model outperformed the two-stage neural baseline in 6 out of 12 language pairs and significantly surpassed traditional Phrase-Based Statistical Machine Translation (PBSMT) in 11 out of 12 pairs. For Marathi-to-Hindi transliteration, the model achieved 59.0% accuracy, significantly higher than the two-stage PBSMT (46.9%).

Bridge Captioning

This task involved generating French captions for images using English as the pivot. The model used MSCOCO images ($X$) and English captions ($Z$) to learn the shared representation, and then decoded into French ($Y$).

Qualitative comparison of French captions generated from images using the Correlational Encoder Decoder, Two-Stage, and Pseudo Im-Fr models.

While the CED model showed competitive performance, it did not consistently beat the two-stage baseline in this task. The authors attribute this to the high complexity of mapping pixels to multiple languages without an attention mechanism. As seen in the qualitative results, the model successfully captures the core subjects of the images (e.g., "un homme," "une girafe," "un bus") but sometimes struggles with complex scene descriptions.

Conclusion

The Correlational Encoder-Decoder architecture demonstrates the feasibility of learning a shared "vectorial interlingua" for pivot-based sequence generation. By jointly optimizing for correlation and generation, the model avoids the pitfalls of cascading systems in transliteration tasks and scales more gracefully toward multi-language systems. Future work will focus on incorporating attention mechanisms to handle more complex multimodal mappings and non-monotonic alignments.

Summary updated

Abstract

Interlingua based Machine Translation (MT) aims to encode multiple languages into a common linguistic representation and then decode sentences in multiple target languages from this representation. In this work we explore this idea in the context of neural encoder decoder architectures, albeit on a smaller scale and without MT as the end goal. Specifically, we consider the case of three languages or modalities X, Z and Y wherein we are interested in generating sequences in Y starting from information available in X. However, there is no parallel training data available between X and Y but, training data is available between X & Z and Z & Y (as is often the case in many real world applications). Z thus acts as a pivot/bridge. An obvious solution, which is perhaps less elegant but works very well in practice is to train a two stage model which first converts from X to Z and then from Z to Y. Instead we explore an interlingua inspired solution which jointly learns to do the following (i) encode X and Z to a common representation and (ii) decode Y from this common representation. We evaluate our model on two tasks: (i) bridge transliteration and (ii) bridge captioning. We report promising results in both these applications and believe that this is a right step towards truly interlingua inspired encoder decoder architectures.
BibTeX
@inproceedings{saha2016correlational,
  author = {Amrita Saha and Mitesh M. Khapra and Sarath Chandar and Janarthanan Rajendran and Kyunghyun Cho},
  title = {{A Correlational Encoder Decoder Architecture for Pivot Based Sequence Generation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
  url = {https://openreview.net/forum?id=BZENlCVSp19},
}
Figure