Abstract
Representing documents by vectors that are independent of language enhances machine translation and multilingual text categorization. We use discriminative training to create a projection of documents from multiple languages into a single translingual vector space. We explore two variants to create these projections: Oriented Principal Component Analysis (OPCA) and Coupled Probabilistic Latent Semantic Analysis (CPLSA). Both of these variants start with a basic model of documents (PCA and PLSA). Each model is then made discriminative by encouraging comparable document pairs to have similar vector representations. We evaluate these algorithms on two tasks: parallel document retrieval for Wikipedia and Europarl documents, and cross-lingual text classification on Reuters. The two discriminative variants, OPCA and CPLSA, significantly outperform their corresponding baselines. The largest differences in performance are observed on the task of retrieval when the documents are only comparable and not parallel. The OPCA method is shown to perform best.
Introduction
As global information exchange accelerates, Natural Language Processing (NLP) systems must increasingly operate across diverse languages. A fundamental challenge is representing documents from different languages in a single, shared mathematical space. Traditionally, researchers have relied on two main strategies:
- Machine Translation (MT): Translating all documents into a pivot language. While accurate, this is computationally intensive and often impractical for large-scale datasets.
- Generative Projections: Unsupervised techniques like Latent Semantic Indexing (LSI) map words to "concepts." However, these methods do not explicitly learn that an English document and its Spanish translation should reside in the same location in the vector space.
This paper proposes a discriminative approach that leverages known pairs of comparable documents to learn a projection where related content is pulled together in a low-dimensional, language-independent space.
Discriminative Projection Methods
The authors introduce two primary algorithms that move beyond simple generative modeling by incorporating supervision from comparable document pairs.
Oriented Principal Component Analysis (OPCA)
OPCA is a discriminative evolution of standard PCA. While PCA identifies directions of maximum variance, OPCA treats the total variance of all documents as the "signal" and the variance between comparable document pairs as "noise." By solving a generalized eigenproblem, OPCA maximizes the signal-to-noise ratio. This results in a projection where documents are well-distributed to preserve meaning, but comparable pairs are kept in close proximity.
Coupled Probabilistic Latent Semantic Analysis (CPLSA)
Building on PLSA topic modeling, CPLSA offers a more flexible approach than standard joint models. Instead of forcing a single topic distribution onto a document pair, CPLSA allows each document to have its own distribution while applying a "posterior regularization" constraint. This constraint penalizes the model if the expected topic proportions for an English document and its Spanish counterpart diverge significantly.
Experimental Evaluation
The methods were evaluated on three datasets representing different levels of translation quality: Europarl (clean parallel text), Wikipedia (noisy comparable articles), and the Multilingual Reuters Collection (for classification).
Key Results
The discriminative models, particularly OPCA, showed significant performance gains over generative baselines. On the challenging Wikipedia dataset, OPCA achieved a 72.6% Top-1 accuracy for document retrieval, outperforming both word-by-word translation (70.3%) and the CL-LSI baseline (53.0%).
In cross-lingual text classification, OPCA approached the accuracy of full machine translation (0.595 vs 0.648) while being approximately 1,000 times faster. This efficiency makes these projection methods ideal for real-time categorization and large-scale search engines.
Conclusion and Future Work
The primary insight of this work is that explicitly training models to recognize comparable documents (discriminative training) is far more effective than unsupervised patterns (generative training). OPCA emerged as the most robust method, partly due to its ability to leverage "anchor words"—proper nouns that appear identically across languages.
Future research directions include addressing vocabulary scaling for OPCA, which currently requires quadratic covariance matrix computations, and exploring hierarchical topic models to improve CPLSA's performance on diverse datasets like Wikipedia. Additionally, extending these bilingual experiments to a truly "massively multilingual" space remains a key challenge for the field.
Summary updated