Abstract
Topic models are a useful tool for analyzing large text collections, but have previously been applied in only monolingual, or at most bilingual, contexts. Meanwhile, massive collections of interlinked documents in dozens of languages, such as Wikipedia, are now widely available, calling for tools that can characterize content in many languages. We introduce a polylingual topic model that discovers topics aligned across multiple languages. We explore the model's characteristics using two large corpora, each with over ten different languages, and demonstrate its usefulness in supporting machine translation and tracking topic trends across languages.
Introduction: The Challenge of Multilingual Analysis
Traditional topic models are usually monolingual. While bilingual models existed prior to this research, they typically relied on "parallel corpora"—direct, sentence-aligned translations like UN proceedings. These models are difficult to scale to many languages because word-to-word alignment becomes exponentially complex as more languages are added.
However, the internet is rich with "comparable" corpora: documents that discuss the same subject but are not direct translations. For example, Wikipedia articles about "The Moon" in English, French, and Persian cover the same core concepts but vary in length, detail, and structure. The authors address the challenge of performing topic-based analysis across these diverse, interlinked collections to track international trends and support machine translation.
Methods: The Polylingual Topic Model (PLTM)
The authors introduce the Polylingual Topic Model (PLTM), a generative model that extends Latent Dirichlet Allocation (LDA) by introducing the concept of a document tuple.
- The Tuple Assumption: A tuple is a set of documents in different languages that are "loosely equivalent" (e.g., Wikipedia articles linked via interlanguage links). PLTM assumes that all documents in a tuple share the exact same topic distribution ($\theta$).
- Language-Specific Topics: While the topic proportions are shared, the content of the topics is language-specific. For a given topic (e.g., "Space Exploration"), the model learns a separate word distribution ($\phi^l$) for every language $l$ in the corpus.
- Generative Process: To generate a tuple, the model first picks a topic distribution. Then, for every word in every document in that tuple, it picks a topic from that distribution and then picks a word from the corresponding language-specific version of that topic.
This approach is significantly simpler than previous bilingual models because it avoids the "alignment bottleneck." It does not require word-level translation; it only requires that the documents are about the same subject.
Datasets and Evaluation
The researchers evaluated PLTM on two massive, multi-lingual datasets:
- EuroParl: Proceedings of the European Parliament in 11 languages. This served as a "gold standard" because these are direct translations.
- Wikipedia: Articles in 12 languages, including diverse scripts and language families (Welsh, German, Greek, English, Farsi, Finnish, French, Hebrew, Italian, Polish, Russian, and Turkish). This represented the more challenging "comparable" text scenario.
Key Insights and Results
- Regularization via Diversity: PLTM often produces better topics than monolingual LDA. The requirement that a topic must be consistent across 11 different languages acts as a form of regularization, filtering out language-specific noise and focusing on meaningful semantic clusters.
- Lexicon Induction: PLTM proved effective at generating bilingual lexicons. By identifying the top words for the same topic in different languages, the model identified correct translations with high precision.
- Document Alignment: The model can find translations in unstructured document collections. By comparing inferred topic distributions, the model correctly identified document translations with ~73% accuracy for documents over 50 words long.
- Scalability: Unlike previous methods that struggled with more than two languages, PLTM handled 12 languages simultaneously, providing a unified semantic space for a diverse global corpus.
Limitations and Future Work
The model struggles with very short or formulaic documents where there isn't enough text to accurately infer a topic distribution. Furthermore, like all LDA-based models, PLTM ignores word order and syntax. Future work could explore ways to align topics with even fewer "glue" documents or by using cross-lingual word embeddings as a prior. Finally, while topics are aligned, the sentiment or cultural perspective on a topic might vary by language, a nuance the current model does not explicitly capture.
Summary updated