Bilingual Autoencoders with Global Descriptors for Modeling Parallel Sentences

Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, Min Zhang

Introduction

Representing sentences as continuous-valued vectors is a cornerstone of modern Natural Language Processing (NLP), particularly for cross-lingual tasks like machine translation and retrieval. Traditional approaches often struggle with a trade-off: complex recursive or recurrent models capture structure but are computationally heavy, while simple Bag-of-Words (BoW) models are scalable but lose nuance by relying solely on the average of word embeddings. This paper, presented at COLING 2016, introduces a "middle ground" that leverages global statistical descriptors within a bilingual autoencoder framework.

The Bilingual Autoencoder with Global Descriptors

The core innovation lies in moving beyond simple averaging. The authors propose using four statistical descriptors—minimum, maximum, average, and standard deviation—extracted from each dimension of the word embeddings in a sentence. These descriptors provide a more comprehensive "global" view of the sentence's semantic features.

Architecture and Constraints

The model consists of two monolingual autoencoders (source and target) that are linked through shared parameters and semantic constraints. As shown in Figure 1, the encoding process takes one set of descriptors (e.g., min and max) and maps them to a hidden latent representation. The decoding process then attempts to reconstruct a different set of descriptors (e.g., average and standard deviation).

The bilingual autoencoder architecture using global descriptors and semantic constraints.

To bridge the gap between languages, the authors implement two key strategies:

  1. Shared Decoding: Both autoencoders use identical decoding parameters ($W^{dec}$), forcing the latent space to be language-agnostic.
  2. Corpus-level Semantic Constraint: Rather than forcing an exact match for every sentence pair—which can be too rigid given the fluidity of translation—the model minimizes the distance between the centers of sentence batches in the latent space. This "relaxed" supervision makes the model more robust to noise and translation variations.

Experimental Results

The model was trained on a large-scale Chinese-English corpus of 2.9 million sentence pairs. Evaluation focused on both semantic similarity and its impact on Statistical Machine Translation (SMT).

Semantic Similarity

In the task of distinguishing parallel sentence pairs from non-parallel ones, the proposed model significantly outperformed the standard Bilingual Bag-of-Words Autoencoder (BBoWAE). As shown in Figure 6, the model achieved an average accuracy of 86.52%, representing a substantial 10% absolute improvement over the baseline.

Performance comparison between the proposed model and the BBoWAE baseline on MT06 and MT08 datasets.

Machine Translation

When integrated into an SMT system, the model provided consistent gains. The best-performing variant achieved an average BLEU score of 28.08, outperforming the baseline by 1.46 points. This demonstrates that the richer semantic representations captured by global descriptors translate directly into better downstream performance.

Conclusion

By incorporating global statistical descriptors and employing a relaxed corpus-level semantic constraint, this bilingual autoencoder offers a scalable and effective way to model parallel sentences. It successfully captures linguistic nuances that simple averaging misses, providing a powerful tool for cross-lingual NLP tasks without the computational burden of more complex architectures.

Summary updated

Abstract

Parallel sentence representations are important for bilingual and cross-lingual tasks in natural language processing. In this paper, we explore a bilingual autoencoder approach to model parallel sentences. We extract sentence-level global descriptors (e.g. min, max) from word embeddings, and construct two monolingual autoencoders over these descriptors on the source and target language. In order to tightly connect the two autoencoders with bilingual correspondences, we force them to share the same decoding parameters and minimize a corpus-level semantic distance between the two languages. Being optimized towards a joint objective function of reconstruction and semantic errors, our bilingual antoencoder is able to learn continuous-valued latent representations for parallel sentences. Experiments on both intrinsic and extrinsic evaluations on statistical machine translation tasks show that our autoencoder achieves substantial improvements over the baselines.
BibTeX
@inproceedings{zhang2016bilingual,
  author = {Biao Zhang and Deyi Xiong and Jinsong Su and Hong Duan and Min Zhang},
  title = {{Bilingual Autoencoders with Global Descriptors for Modeling Parallel Sentences}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure