Abstract
Topic modeling has traditionally relied on simple priors that struggle to incorporate high-dimensional or complex metadata. We present Deep Dirichlet Multinomial Regression (dDMR), an extension of the Dirichlet Multinomial Regression (DMR) model that replaces log-linear priors with deep neural networks. This architectural shift allows the model to automatically learn feature representations from diverse metadata types, such as images and thousands of sparse tags. Experimental results across multiple datasets demonstrate that dDMR provides better model fit, significantly faster convergence, and improved topic quality as judged by human evaluators.
Introduction
Latent Dirichlet Allocation (LDA) remains a cornerstone of unsupervised text analysis, yet it often fails to leverage the rich metadata—authors, timestamps, or images—that frequently accompanies modern document collections. While Dirichlet Multinomial Regression (DMR) was developed to address this by using document features to influence topic priors, its reliance on a simple log-linear mapping limits its scalability and effectiveness with high-dimensional data. Standard DMR often requires extensive manual feature engineering and is prone to overfitting when faced with thousands of features.
To overcome these limitations, we introduce dDMR, which utilizes a deep neural network to map raw document features to latent topic representations. This approach enables the model to capture non-linear relationships and automatically learn useful representations from complex metadata.
Methods
The core innovation of dDMR is the replacement of the linear mapping in standard DMR with a deep neural network $f$.
The Generative Process
The model follows a structured generative story:
- Neural Mapping: A function $f$ (typically a Multi-Layer Perceptron) transforms a document's raw features $\alpha_m$ into a latent representation $\tilde{\theta}_m$.
- Topic Prior: This representation serves as the hyperparameter for the Dirichlet distribution, which governs the document's topic mixture $\theta_m$.
- Joint Optimization: The model parameters (neural network weights) and latent topics are trained simultaneously. We employ collapsed Gibbs sampling for topic estimation and backpropagation with gradient ascent to update the neural network based on the likelihood of the observed text given the current topics.
By incorporating non-linear activations like ReLU, dDMR can model intricate dependencies between metadata and text that are inaccessible to linear frameworks.
Experimental Evaluation
We evaluated dDMR on three distinct datasets to prove its flexibility:
- New York Times (NYT): Articles with 2,424 unique descriptor tags, testing high-dimensional sparse label handling.
- Amazon Reviews: Reviews paired with 4,096-dimensional image features extracted via a pre-trained CNN.
- Reddit: Short-text comments paired with subreddit IDs, where metadata is vital for disambiguation.
Results and Discussion
Convergence and Model Fit
A primary advantage of dDMR is its efficiency. Supervision via metadata acts as a catalyst for training. As shown in the convergence plots, dDMR often achieves lower held-out perplexity in 1,000 iterations than unsupervised LDA does in 10,000.
Robustness and Interpretability
Unlike standard DMR, which is sensitive to L1/L2 regularization tuning, dDMR demonstrates greater robustness across various learning rates and penalties. Furthermore, human evaluation via Amazon Mechanical Turk indicated a strong preference for dDMR-generated topics. When presented with a product image, users found dDMR's topic words significantly more descriptive of the visual content than those from baseline models.
Conclusion
Deep Dirichlet Multinomial Regression successfully bridges the gap between traditional probabilistic topic modeling and deep learning. By allowing neural networks to learn the influence of metadata on topic distributions, dDMR offers a more flexible, robust, and performant solution for analyzing metadata-rich document collections. Future work may explore more specialized architectures, such as RNNs for sequential metadata or deeper CNNs for direct image processing within the topic modeling loop.
Summary updated