Abstract
Cross-lingual topic modeling aims to uncover shared semantic themes across languages. Several methods have been proposed to address this problem, leveraging both traditional and neural approaches. While previous methods have achieved some improvements in topic diversity, they often struggle to ensure high topic coherence and consistent alignment across languages. We propose XTRA (Cross-Lingual Topic Modeling with Topic and Representation Alignments), a novel framework that unifies Bag-of-Words modeling with multilingual embeddings. XTRA introduces two core components: (1) representation alignment, aligning document-topic distributions via contrastive learning in a shared semantic space; and (2) topic alignment, projecting topic-word distributions into the same space to enforce cross-lingual consistency. This dual mechanism enables XTRA to learn topics that are interpretable (coherent and diverse) and well-aligned across languages. Experiments on multilingual corpora confirm that XTRA significantly outperforms strong baselines in topic coherence, diversity, and alignment quality. Code and reproducible scripts are available at https://github.com/tienphat140205/XTRA.
Introduction
Cross-lingual Topic Modeling (CLTM) is essential for discovering shared semantic themes across different languages—for example, identifying that a set of English words about "piano" and "concert" describes the same theme as a set of Chinese words for "singer" and "guitar." However, existing CLTMs face three major hurdles: lexical coverage limitations due to incomplete bilingual dictionaries, topic collapse where neural models produce redundant topics, and alignment gaps between word distributions ($\beta$) and document-level topic proportions ($\theta$).
Methods
XTRA is built on a Variational Autoencoder (VAE) architecture, incorporating an Enhanced Cross-Lingual Encoder and a Dual-Alignment Mechanism to bridge the gap between languages without requiring parallel corpora.
The framework consists of several key innovations:
- Enhanced Encoder: Lightweight MLPs project Bag-of-Words (BoW) inputs into a language-agnostic space handled by a shared encoder, preventing language-specific biases.
- Representation ($\theta$) Alignment: The model uses InfoNCE loss to align latent topic proportions ($\theta$) with high-quality semantic embeddings from a Pre-trained Language Model (PLM). Clustering Contrastive Alignment further ensures documents in the same semantic cluster share similar topic distributions across languages.
- Topic ($\beta$) Alignment: High-dimensional topic-word distributions are projected into a lower-dimensional semantic space. A contrastive loss pulls corresponding cross-lingual topic vectors together while pushing non-corresponding ones apart, ensuring "Topic 1" in English matches "Topic 1" in Chinese.
- Cluster-based Prior: A cluster-based Gaussian prior replaces the standard Gaussian prior to better capture the corpus's underlying semantic structure.
Experimental Results
The authors evaluated XTRA on three benchmarks: EC News (English/Chinese), Amazon Review (English/Chinese), and Rakuten Amazon (Japanese/English).
XTRA significantly outperformed state-of-the-art baselines like InfoCTM and NMTM across several metrics:
- Topic Quality (TQ): On EC News, XTRA achieved a TQ of 0.075, nearly doubling the performance of neural baselines.
- Topic Uniqueness (TU): XTRA maintained near-perfect diversity scores (up to 0.993), indicating it successfully avoided redundant topics.
- Downstream Utility: In cross-lingual classification tasks, XTRA showed a massive accuracy boost. On EC News, the cross-lingual accuracy for Chinese (ZH-C) reached 0.582, compared to 0.420 for NMTM.
- Qualitative Evaluation: LLM-based evaluations (using GPT-4o) confirmed that XTRA’s topics were more coherent and better aligned than those of its competitors, with fewer "noisy" or irrelevant words.
Conclusion
XTRA addresses the fundamental limitations of existing CLTMs by aligning both document-level and topic-level distributions in a shared semantic space. By combining the interpretability of Bag-of-Words with the deep contextual knowledge of Transformers, it achieves superior topic coherence and alignment. While the model requires pre-specified parameters and involves computational overhead for PLM embeddings, it represents a significant step forward in robust cross-lingual modeling.
Summary updated