OTLDA: A Geometry-aware Optimal Transport Approach for Topic Modeling

Viet Huynh, He Zhao, Dinh PhungNeurIPS 2020 · OpenReview

Overview

Traditional topic models like Latent Dirichlet Allocation (LDA) treat words as discrete, independent tokens. This approach ignores the semantic reality that words like "bicycle" and "bike" are closely related. Because LDA relies heavily on exact word co-occurrences, it struggles when related words don't appear in the same document—a "semantic blindness" that is especially problematic for short texts like tweets or documents with rare synonyms.

To bridge the gap between statistical word counts and semantic meaning, researchers introduce OTLDA (Optimal Transport Latent Dirichlet Allocation). By replacing traditional loss functions with the Wasserstein distance and leveraging the geometry of word embeddings, OTLDA produces highly coherent topics and excels at modeling short, sparse texts where traditional models fail.

The Optimal Transport Approach

OTLDA re-imagines topic modeling through the lens of Optimal Transport (OT). Instead of viewing a document merely as a bag of words, OTLDA treats it as a probability distribution in a geometric space.

Geometric interpretation of topic modeling on a probability simplex
  • Geometric Perspective: The model measures the "distance" between a document and a topic using the Wasserstein distance (often called the Earth Mover's Distance).
  • Semantic Cost Function: OTLDA uses the distance between pre-trained word embeddings (e.g., the Euclidean distance between word2vec vectors) as the "cost" to move probability mass. This naturally encourages the model to group semantically similar words together, creating a "semantic gravity" that pulls synonyms into the same topic even if they rarely co-occur.
  • Polytope Interpretation: Learning topics is conceptually equivalent to finding a geometric shape (a polytope) that encloses document observations on a simplex. OTLDA warps this simplex based on actual word meanings rather than just statistical frequency.
  • Optimization: The framework uses an efficient alternating optimization algorithm, updating document-topic proportions via fast Sinkhorn iterations and updating the topics themselves by calculating Optimal Transport Barycenters. Crucially, OTLDA explicitly incorporates document length into its loss function, allowing it to handle heterogeneous corpora effectively.

Key Results and Performance

The model was evaluated across four distinct datasets to test its versatility: standard newsgroup posts (20NG), a large-scale Wikipedia corpus, a subset of short newsgroup posts (20NGshort), and a highly sparse Twitter dataset (averaging just 8 words per tweet).

Topic coherence metrics across different models and datasets
  • Superior Topic Coherence: OTLDA achieved the highest scores in both intrinsic (UMass) and extrinsic (UCI) coherence metrics across almost all datasets. The performance gap is particularly striking on short texts. For example, on the Twitter dataset, OTLDA's UCI score was 0.162, vastly outperforming the next best model, ETM (-1.149).
  • Robustness to Sparsity: While traditional LDA's performance collapses when documents are only a few words long, OTLDA maintains high topic coherence by leaning on the pre-trained geometry of word embeddings. Qualitative analysis also showed that OTLDA identifies more specific, descriptive words compared to other neural topic models.
  • Downstream Accuracy: Beyond coherence, the learned topics provide better representations for downstream tasks. In a document classification task on the 20NG dataset, OTLDA achieved an accuracy of 47.8%, outperforming the popular Embedded Topic Model (ETM).
  • Scalability: Some competing Wasserstein-based models (like LFTM and WNMF) are too computationally heavy to run on large datasets. OTLDA processed the 100,000-document Wikipedia dataset efficiently while maintaining superior coherence.

Limitations and Future Directions

While powerful, OTLDA's performance is sensitive to its regularization parameter ($\lambda$). Empirical results suggest that regular-length documents require a larger $\lambda$ (around 50), whereas short documents perform better with a very small $\lambda$ (around 0.05).

Currently, the framework relies on static, pre-trained word embeddings and operates in a strictly unsupervised manner. Future work could explore jointly fine-tuning these embeddings during the topic modeling process, or extending the OTLDA framework to supervised settings and dynamic environments where topics evolve over time.

Summary updated

Abstract

We present an optimal transport framework for learning topics from textual data. While the celebrated Latent Dirichlet allocation (LDA) topic model and its variants have been applied to many disciplines, they mainly focus on word-occurrences and neglect to incorporate semantic regularities in language. Even though recent works have tried to exploit the semantic relationship between words to bridge this gap, however, these models which are usually extensions of LDA or Dirichlet Multinomial mixture (DMM) are tailored to deal effectively with either regular or short documents. The optimal transport distance provides an appealing tool to incorporate the geometry of word semantics into it. Moreover, recent developments on efficient computation of optimal transport distance also promote its application in topic modeling. In this paper we ground on optimal transport theory to naturally exploit the geometric structures of semantically related words in embedding spaces which leads to more interpretable learned topics. Comprehensive experiments illustrate that the proposed framework outperforms competitive approaches in terms of topic coherence on assorted text corpora which include both long and short documents. The representation of learned topic also leads to better accuracy on classification downstream tasks, which is considered as an extrinsic evaluation.
BibTeX
@inproceedings{huynh2020otlda,
  author = {Viet Huynh and He Zhao and Dinh Phung},
  title = {{OTLDA: A Geometry-aware Optimal Transport Approach for Topic Modeling}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=1yF7xCIV1W},
}
Figure