TL;DR
This paper introduces Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network (SawETM), a novel deep generative model for topic modeling. SawETM addresses the limitation of existing hierarchical topic models by explicitly capturing dependencies and semantic similarities between topics across different layers, leading to more interpretable topics and better document representations.
What Problem Does This Paper Address?
Traditional topic models like Latent Dirichlet Allocation (LDA) represent documents as mixtures of independent topics. While hierarchical extensions like Gamma Belief Networks (GBNs) introduce topic hierarchies, they often assume independence between topics at the same and different layers. This assumption limits their ability to capture nuanced thematic structures and semantic relationships. Furthermore, existing neural topic models (NTMs) often struggle with deep architectures due to issues like posterior collapse and approximating sparse, non-negative document representations. SawETM aims to overcome these limitations by proposing a deep hierarchical topic model that explicitly models inter-topic dependencies and uses a robust inference network for better representation learning.
Methods
The core innovation of SawETM lies in its Sawtooth Connection (SC) technique. This method factorizes the topic matrix at each layer into a factor loading matrix and a topic embedding matrix. Crucially, the transpose of the factor loading matrix from layer l is used as the factor loading matrix for layer l-1. This "sawtooth" factorization creates a dependency structure that links topics across adjacent layers, allowing knowledge to flow from lower to higher layers.
For inference, SawETM employs a deep hierarchical variational auto-encoder (VAE). To combat posterior collapse in deep VAEs, it incorporates a skip-connected deterministic upward path alongside the stochastic path. This deterministic path helps propagate information more effectively, stabilizing training. The model also utilizes a Weibull distribution for approximating the posterior of latent variables, which is reparameterizable and better suited for non-negative document representations compared to Gaussian distributions.
Datasets
The experiments were conducted on four real-world datasets:
- R8: A subset of Reuters 21578, used for document clustering.
- 20Newsgroups (20NG): A standard benchmark for text classification, also used for document clustering.
- Reuters Corpus Volume I (RCV1): A large corpus for text analysis.
- PG-19: A dataset extracted from Project Gutenberg, used for topic modeling experiments.
Key Insights and Ideas
- Dependencies Matter: Explicitly modeling dependencies between topics across hierarchical layers significantly improves topic interpretability and document representation quality.
- Sawtooth for Knowledge Flow: The "Sawtooth Connection" provides an elegant mechanism for transferring learned semantic information from lower to higher topic layers, addressing sparsity and improving generalization.
- Deep VAEs Can Work for Topics: By carefully designing a hierarchical VAE with deterministic skip connections, it's possible to train deep neural topic models effectively and avoid common pitfalls like posterior collapse.
- Weibull for Non-Negative Data: The Weibull distribution proves to be a more suitable choice than Gaussian for approximating the posterior of non-negative latent variables in topic models.
Key Results
- Perplexity: SawETM achieves comparable or better per-heldout-word perplexity than state-of-the-art models, especially as the number of layers increases. It performs particularly well on larger datasets where deep structures are beneficial.
- Topic Quality: SawETM demonstrates competitive topic quality (a combination of coherence and diversity), often outperforming other neural topic models and approaching the performance of traditional Gibbs-based models.
- Document Clustering: SawETM extracts more expressive latent document representations, leading to state-of-the-art performance in document clustering tasks on the 20NG and R8 datasets, achieving the highest AC and NMI scores.
- Interpretability: Qualitative analysis shows that SawETM learns meaningful word and topic embeddings, and its hierarchical structure allows for the visualization of interpretable topic taxonomies.
Limitations and Future Work
- Computational Cost: While SawETM is optimized via SGD and is scalable, training deep hierarchical models can still be computationally intensive.
- Hyperparameter Sensitivity: Like many deep learning models, SawETM's performance may be sensitive to hyperparameter choices, although the paper mentions standard settings and optimization strategies.
- Future Work: The authors suggest that SawETM's ability to capture dependencies and semantic similarities could be extended to other areas, such as dynamic topic modeling or cross-lingual topic modeling. The deep hierarchical structure also opens avenues for more complex document representation learning.
Summary updated