Abstract
Cross-language learning allows one to use training data from one language to build models for another language. Many traditional approaches require word-level alignment sentences from parallel corpora; in this paper, we define a general bilingual training objective function requiring sentence-level parallel corpus only. We propose a variational autoencoding approach for training bilingual word embeddings. The variational model introduces a continuous latent variable to explicitly model the underlying semantics of the parallel sentence pairs and to guide the generation of the sentence pairs. Our model restricts the bilingual word embeddings to represent words in exactly the same continuous vector space. Empirical results on the task of cross-lingual document classification have shown that our method is effective.
Introduction
To perform Natural Language Processing (NLP) tasks across languages—such as training a sentiment analyzer in English and applying it to German—we need a way to represent words from both languages in the same mathematical space. Traditional methods for creating these "cross-lingual embeddings" often rely on word-level alignments (knowing exactly which word in sentence A maps to which word in sentence B), which are expensive and difficult to obtain.
The researchers propose BiVAE (Bilingual Variational Autoencoder), a generative model that learns to map two different languages into a single, shared semantic vector space. By treating parallel sentence pairs as different "observations" of the same underlying concept, the model induces high-quality cross-lingual word embeddings using only sentence-level alignments, outperforming several established baselines in document classification.
The BiVAE Framework
The core idea of BiVAE is based on a generative assumption: there exists a latent (hidden) semantic variable $z$ that represents the "meaning" of a sentence, and this $z$ independently generates the equivalent sentences in both Language A and Language B.
The architecture consists of three main components:
- Neural Sentence Encoder: Uses bidirectional LSTMs to process parallel sentences, capturing context from both directions to create a robust representation ($h$) of the sentence pair.
- Variational Inference Network: Predicts the parameters (mean $\mu$ and variance $\log\sigma^2$) of a Gaussian distribution in the latent space. A specific "meaning" $z'$ is then sampled using the reparameterization trick.
- Bag-of-Words (BoW) Decoder: Unlike standard VAEs that might use an LSTM to generate sentences word-by-word, BiVAE uses a softmax-based BoW decoder. This choice accelerates training and prevents the model from getting bogged down in language-specific syntax, forcing it to focus on global semantic content.
The weights of the decoder effectively become the word embeddings. Because both languages are reconstructed from the same latent variable, their embeddings are naturally pushed into the same continuous vector space.
Experimental Results
The model was trained on the Europarl v7 parallel corpus (English-German) and evaluated on the Reuters RCV1/RCV2 multilingual corpora for Cross-Lingual Document Classification (CLDC). In this task, a classifier is trained on labeled data in one language and tested on another.
As shown in the results table, BiVAE achieved 91.0% accuracy on English-to-German ($en \to de$) and 80.4% on German-to-English ($de \to en$). It outperformed several well-known models like BiCVM (83.7%) and BilBOWA (86.5%) and performed comparably to the state-of-the-art CLSim (92.7%), despite BiVAE using only parallel data while other methods often benefit from additional data structures.
Qualitative analysis further supports these findings. Visualization using t-SNE showed that semantically equivalent pairs (e.g., "children" and "kinder") were mapped to nearly identical coordinates. Additionally, nearest neighbor analysis revealed that the model successfully captures semantic clusters across languages, such as matching "oil" in English with "Öl" and "erdöl" in German.
Conclusion
BiVAE provides a unified probabilistic framework for inducing cross-lingual embeddings without the need for explicit word-level alignments. By using a "bottleneck" latent variable and a Bag-of-Words decoder, the model prioritizes semantic content over syntactic nuances. Future work may explore leveraging monolingual data to further refine these representations or incorporating more sophisticated decoders to generate synthetic parallel data.
Summary updated