Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network

Zhibin Duan, Dongsheng Wang, Bo Chen, Chaojie Wang, Wenchao Chen, Yewen Li, Jie Ren, Mingyuan Zhou

TL;DR

This paper introduces Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network (SawETM), a novel deep generative model for topic modeling. SawETM addresses the limitation of existing hierarchical topic models by explicitly capturing dependencies and semantic similarities between topics across different layers, leading to more interpretable topics and better document representations.

What Problem Does This Paper Address?

Traditional topic models like Latent Dirichlet Allocation (LDA) represent documents as mixtures of independent topics. While hierarchical extensions like Gamma Belief Networks (GBNs) introduce topic hierarchies, they often assume independence between topics at the same and different layers. This assumption limits their ability to capture nuanced thematic structures and semantic relationships. Furthermore, existing neural topic models (NTMs) often struggle with deep architectures due to issues like posterior collapse and approximating sparse, non-negative document representations. SawETM aims to overcome these limitations by proposing a deep hierarchical topic model that explicitly models inter-topic dependencies and uses a robust inference network for better representation learning.

Methods

The core innovation of SawETM lies in its Sawtooth Connection (SC) technique. This method factorizes the topic matrix at each layer into a factor loading matrix and a topic embedding matrix. Crucially, the transpose of the factor loading matrix from layer l is used as the factor loading matrix for layer l-1. This "sawtooth" factorization creates a dependency structure that links topics across adjacent layers, allowing knowledge to flow from lower to higher layers.

For inference, SawETM employs a deep hierarchical variational auto-encoder (VAE). To combat posterior collapse in deep VAEs, it incorporates a skip-connected deterministic upward path alongside the stochastic path. This deterministic path helps propagate information more effectively, stabilizing training. The model also utilizes a Weibull distribution for approximating the posterior of latent variables, which is reparameterizable and better suited for non-negative document representations compared to Gaussian distributions.

Datasets

The experiments were conducted on four real-world datasets:

  • R8: A subset of Reuters 21578, used for document clustering.
  • 20Newsgroups (20NG): A standard benchmark for text classification, also used for document clustering.
  • Reuters Corpus Volume I (RCV1): A large corpus for text analysis.
  • PG-19: A dataset extracted from Project Gutenberg, used for topic modeling experiments.

Key Insights and Ideas

  • Dependencies Matter: Explicitly modeling dependencies between topics across hierarchical layers significantly improves topic interpretability and document representation quality.
  • Sawtooth for Knowledge Flow: The "Sawtooth Connection" provides an elegant mechanism for transferring learned semantic information from lower to higher topic layers, addressing sparsity and improving generalization.
  • Deep VAEs Can Work for Topics: By carefully designing a hierarchical VAE with deterministic skip connections, it's possible to train deep neural topic models effectively and avoid common pitfalls like posterior collapse.
  • Weibull for Non-Negative Data: The Weibull distribution proves to be a more suitable choice than Gaussian for approximating the posterior of non-negative latent variables in topic models.

Key Results

  • Perplexity: SawETM achieves comparable or better per-heldout-word perplexity than state-of-the-art models, especially as the number of layers increases. It performs particularly well on larger datasets where deep structures are beneficial.
  • Topic Quality: SawETM demonstrates competitive topic quality (a combination of coherence and diversity), often outperforming other neural topic models and approaching the performance of traditional Gibbs-based models.
  • Document Clustering: SawETM extracts more expressive latent document representations, leading to state-of-the-art performance in document clustering tasks on the 20NG and R8 datasets, achieving the highest AC and NMI scores.
  • Interpretability: Qualitative analysis shows that SawETM learns meaningful word and topic embeddings, and its hierarchical structure allows for the visualization of interpretable topic taxonomies.

Limitations and Future Work

  • Computational Cost: While SawETM is optimized via SGD and is scalable, training deep hierarchical models can still be computationally intensive.
  • Hyperparameter Sensitivity: Like many deep learning models, SawETM's performance may be sensitive to hyperparameter choices, although the paper mentions standard settings and optimization strategies.
  • Future Work: The authors suggest that SawETM's ability to capture dependencies and semantic similarities could be extended to other areas, such as dynamic topic modeling or cross-lingual topic modeling. The deep hierarchical structure also opens avenues for more complex document representation learning.

Summary updated

Abstract

Hierarchical topic models such as the gamma belief network (GBN) have delivered promising results in mining multi-layer document representations and discovering interpretable topic taxonomies. However, they often assume in the prior that the topics at each layer are independently drawn from the Dirichlet distribution, ignoring the dependencies between the topics both at the same layer and across different layers. To relax this assumption, we propose sawtooth factorial topic embedding guided GBN, a deep generative model of documents that captures the dependencies and semantic similarities between the topics in the embedding space. Specifically, both the words and topics are represented as embedding vectors of the same dimension. The topic matrix at a layer is factorized into the product of a factor loading matrix and a topic embedding matrix, the transpose of which is set as the factor loading matrix of the layer above. Repeating this particular type of factorization, which shares components between adjacent layers, leads to a structure referred to as sawtooth factorization. An auto-encoding variational inference network is constructed to optimize the model parameter via stochastic gradient descent. Experiments on big corpora show that our models outperform other neural topic models on extracting deeper interpretable topics and deriving better document representations.
BibTeX
@inproceedings{duan2021sawtooth,
  author = {Zhibin Duan and Dongsheng Wang and Bo Chen and Chaojie Wang and Wenchao Chen and Yewen Li and Jie Ren and Mingyuan Zhou},
  title = {{Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=SeZcdyFh6sM},
}

Loading related content...

Figure