Modeling Text using the Continuous Space Topic Model with Pre-Trained Word Embeddings

Seiichi Inoue, Taichi Aida, Mamoru Komachi, Manabu AsaiACL 2021

TL;DR

This paper introduces a novel topic modeling approach by integrating pre-trained word embeddings into the Continuous Space Topic Model (CSTM). This enhancement significantly improves performance in terms of perplexity and convergence speed, and leads to more effective document representations for classification tasks.

What Problem Does This Paper Address?

Traditional topic models like Latent Dirichlet Allocation (LDA) often struggle to capture the nuanced semantic relationships between words within a document. While the Continuous Space Topic Model (CSTM) offers more flexibility by directly modeling word probabilities based on latent coordinates, its original formulation treats these word embeddings as free parameters. This makes the model computationally expensive to train and limits its ability to fully capture word semantics, as it relies solely on word co-occurrence frequencies. The paper aims to address this by leveraging the rich semantic information already encoded in pre-trained word embeddings to improve the CSTM's efficiency and effectiveness.

Methods

The core innovation of this work is the integration of pre-trained word embeddings into the CSTM framework. Specifically, the authors use the word2vec algorithm to learn word embeddings from a large corpus. These learned embeddings, representing word semantics, are then used as the latent coordinates for words within the CSTM.

Here's a breakdown of the key methodological aspects:

  • Pre-trained Word Embeddings: The study utilizes the Continuous Bag-of-Words (CBOW) model from word2vec to generate word embeddings. These embeddings are then centered and normalized to improve their quality.
  • Continuous Space Topic Model (CSTM): The CSTM models documents without intermediate latent topics, instead using latent coordinates for words and a Gaussian process to represent document meaning. This allows for dynamic adjustment of word distributions per document.
  • Integration: The pre-trained word embeddings are directly plugged into the CSTM as the word latent coordinates (φ(w)). This replaces the need to learn these coordinates from scratch within the CSTM, which was a computationally intensive part of the original model.
  • Model Estimation: The paper employs a random walk Metropolis-Hastings (MH) algorithm for model estimation, a standard technique for complex probabilistic models.

Datasets

The authors evaluated their proposed model on three corpora:

  • NIPS: A dataset of Neural Information Processing Systems papers (English).
  • CSJ: Corpus of Spontaneous Japanese (Japanese).
  • Mainichi: A subset of the Mainichi Newspaper corpus (Japanese).

For Japanese corpora, preprocessing was done using MeCab with IPADic. Words with a frequency less than five were excluded from the training data.

Key Insights and Ideas

  • Leveraging Pre-trained Semantics: The paper demonstrates that pre-trained word embeddings are a powerful source of prior knowledge for topic models. By injecting these semantic representations, the model gains a head start in understanding word relationships.
  • Efficiency Boost: Integrating pre-trained embeddings dramatically speeds up the training process. The proposed model converges in significantly fewer iterations compared to the baseline CSTM.
  • Improved Document Representation: The learned document representations are shown to be more effective for downstream tasks like document classification, outperforming the baseline CSTM.
  • Qualitative Understanding: Visualizations of the embedding space reveal that the proposed model places words related to a document's topic more accurately, enhancing the interpretability and utility of the learned representations.

Key Results

  • Intrinsic Evaluation (Perplexity): The proposed model consistently outperformed the baseline CSTM and the Embedded Topic Model (ETM) across all three corpora in terms of perplexity, indicating better predictive power.
  • Convergence Speed: The proposed model converged in less than ten iterations, while the baseline CSTM required fifty to a hundred iterations, showcasing a substantial speedup.
  • Extrinsic Evaluation (Document Classification):
    • When using only document vectors, the proposed model significantly outperformed the CSTM but was slightly inferior to word2vec.
  • However, when combining the proposed model's document vector with word2vec's average word vectors, the accuracy was better than word2vec alone, though the difference was not statistically significant.
  • In a detailed analysis on the CSJ corpus, the proposed model achieved a classification accuracy of 0.866, significantly outperforming the CSTM (0.704). When combined with word2vec, the accuracy further improved to 0.928.

Limitations and Future Work

  • Hamiltonian MCMC: The authors mention that while they attempted Hamiltonian MCMC for estimation, they ultimately used the random walk Metropolis-Hastings algorithm due to computational cost and the need for numerical differentiation. Future work could explore more efficient estimation methods.
  • Contextualized Embeddings: The paper suggests exploring the use of contextualized word embeddings, such as those from ELMo or BERT, in their proposed model. This could further enhance the model's ability to capture nuanced word meanings in different contexts.
  • Further Optimization: The authors plan to investigate better ways of estimating the model, including optimization techniques.

Summary updated

BibTeX
@inproceedings{inoue2021modeling,
  author = {Seiichi Inoue and Taichi Aida and Mamoru Komachi and Manabu Asai},
  title = {{Modeling Text using the Continuous Space Topic Model with Pre-Trained Word Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure