Introduction
Representing sentences as continuous-valued vectors is a cornerstone of modern Natural Language Processing (NLP), particularly for cross-lingual tasks like machine translation and retrieval. Traditional approaches often struggle with a trade-off: complex recursive or recurrent models capture structure but are computationally heavy, while simple Bag-of-Words (BoW) models are scalable but lose nuance by relying solely on the average of word embeddings. This paper, presented at COLING 2016, introduces a "middle ground" that leverages global statistical descriptors within a bilingual autoencoder framework.
The Bilingual Autoencoder with Global Descriptors
The core innovation lies in moving beyond simple averaging. The authors propose using four statistical descriptors—minimum, maximum, average, and standard deviation—extracted from each dimension of the word embeddings in a sentence. These descriptors provide a more comprehensive "global" view of the sentence's semantic features.
Architecture and Constraints
The model consists of two monolingual autoencoders (source and target) that are linked through shared parameters and semantic constraints. As shown in Figure 1, the encoding process takes one set of descriptors (e.g., min and max) and maps them to a hidden latent representation. The decoding process then attempts to reconstruct a different set of descriptors (e.g., average and standard deviation).
To bridge the gap between languages, the authors implement two key strategies:
- Shared Decoding: Both autoencoders use identical decoding parameters ($W^{dec}$), forcing the latent space to be language-agnostic.
- Corpus-level Semantic Constraint: Rather than forcing an exact match for every sentence pair—which can be too rigid given the fluidity of translation—the model minimizes the distance between the centers of sentence batches in the latent space. This "relaxed" supervision makes the model more robust to noise and translation variations.
Experimental Results
The model was trained on a large-scale Chinese-English corpus of 2.9 million sentence pairs. Evaluation focused on both semantic similarity and its impact on Statistical Machine Translation (SMT).
Semantic Similarity
In the task of distinguishing parallel sentence pairs from non-parallel ones, the proposed model significantly outperformed the standard Bilingual Bag-of-Words Autoencoder (BBoWAE). As shown in Figure 6, the model achieved an average accuracy of 86.52%, representing a substantial 10% absolute improvement over the baseline.
Machine Translation
When integrated into an SMT system, the model provided consistent gains. The best-performing variant achieved an average BLEU score of 28.08, outperforming the baseline by 1.46 points. This demonstrates that the richer semantic representations captured by global descriptors translate directly into better downstream performance.
Conclusion
By incorporating global statistical descriptors and employing a relaxed corpus-level semantic constraint, this bilingual autoencoder offers a scalable and effective way to model parallel sentences. It successfully captures linguistic nuances that simple averaging misses, providing a powerful tool for cross-lingual NLP tasks without the computational burden of more complex architectures.
Summary updated