Abstract
Interlingua-based Machine Translation (MT) aims to encode multiple languages into a common linguistic representation and then decode sentences in multiple target languages from this representation. In this work, we explore this idea in the context of neural encoder-decoder architectures, albeit on a smaller scale and without MT as the end goal. Specifically, we consider the case of three languages or modalities $X$, $Z$, and $Y$ wherein we are interested in generating sequences in $Y$ starting from information available in $X$. However, there is no parallel training data available between $X$ and $Y$, but training data is available between $X$ & $Z$ and $Z$ & $Y$. $Z$ thus acts as a pivot/bridge. Instead of a two-stage model, we explore an interlingua-inspired solution which jointly learns to encode $X$ and $Z$ to a common representation and decode $Y$ from this common representation. We evaluate our model on bridge transliteration and bridge captioning, reporting promising results.
Introduction
In many real-world sequence generation tasks, direct parallel training data between a source domain ($X$) and a target domain ($Y$) is unavailable. For instance, translating between two low-resource languages or generating captions in a specific language for images often lacks direct supervision. However, data is frequently available between these domains and a common "pivot" domain ($Z$), such as English.
The standard workaround is a two-stage system: translate from $X$ to $Z$, then translate that output into $Y$. While functional, this "cascading" approach is prone to error propagation from the first stage and fails to learn a truly unified representation of the underlying meaning. This paper proposes a Correlational Encoder-Decoder (CED) architecture that moves toward an interlingua—a common linguistic representation where any language can be encoded into a shared space and decoded into any other.
The Correlational Encoder-Decoder Architecture
The proposed model aims to jointly learn representations for the source and pivot domains while training a decoder for the target domain.
The architecture consists of two encoders (for $X$ and $Z$) and one decoder (for $Y$). The training process involves two primary objectives:
- Correlation Maximization: A correlation loss forces the encoders for $X$ and $Z$ to produce highly similar latent vectors for parallel $(X, Z)$ pairs. This ensures that "cat" in $X$ and "cat" in $Z$ map to the same region in the shared latent space.
- Cross-Entropy Minimization: The decoder is trained to generate the correct sequence in $Y$ using the representation provided by the $Z$ encoder from parallel $(Z, Y)$ pairs.
Because the $X$ and $Z$ representations are forced to be similar, at inference time, the $X$ encoder can be plugged directly into the $Y$ decoder. The model generates $Y$ from $X$ without ever actually "stepping through" the pivot language $Z$.
Experimental Evaluation
The researchers evaluated the CED model on two distinct tasks: bridge transliteration and bridge captioning.
Bridge Transliteration
Using the NEWS 2012 dataset, the model performed transliteration between four Indian languages (Hindi, Kannada, Tamil, and Marathi) using English as the pivot. The CED model outperformed the two-stage neural baseline in 6 out of 12 language pairs and significantly surpassed traditional Phrase-Based Statistical Machine Translation (PBSMT) in 11 out of 12 pairs. For Marathi-to-Hindi transliteration, the model achieved 59.0% accuracy, significantly higher than the two-stage PBSMT (46.9%).
Bridge Captioning
This task involved generating French captions for images using English as the pivot. The model used MSCOCO images ($X$) and English captions ($Z$) to learn the shared representation, and then decoded into French ($Y$).
While the CED model showed competitive performance, it did not consistently beat the two-stage baseline in this task. The authors attribute this to the high complexity of mapping pixels to multiple languages without an attention mechanism. As seen in the qualitative results, the model successfully captures the core subjects of the images (e.g., "un homme," "une girafe," "un bus") but sometimes struggles with complex scene descriptions.
Conclusion
The Correlational Encoder-Decoder architecture demonstrates the feasibility of learning a shared "vectorial interlingua" for pivot-based sequence generation. By jointly optimizing for correlation and generation, the model avoids the pitfalls of cascading systems in transliteration tasks and scales more gracefully toward multi-language systems. Future work will focus on incorporating attention mechanisms to handle more complex multimodal mappings and non-monotonic alignments.
Summary updated