Abstract
Dynamic topic modeling is a well established tool for capturing the temporal dynamics of the topics of a corpus. In this work, we develop a scalable dynamic topic model by utilizing the correlation among the words in the vocabulary. By correlating previously independent temporal processes for words, our new model allows us to reliably estimate the topic representations containing less frequent words. We develop an amortised variational inference method with self-normalised importance sampling approximation to the word distribution that dramatically reduces the computational complexity and the number of variational parameters in order to handle large vocabularies. With extensive experiments on text datasets, we show that our method significantly outperforms the previous works by modeling word correlations, and it is able to handle real world data with a large vocabulary which could not be processed by previous continuous dynamic topic models.
Introduction
Topic modeling is a foundational technique for uncovering latent themes within large document collections. Dynamic Topic Modeling (DTM) extends this by tracking how these themes evolve over time. However, traditional DTMs face two significant challenges: the Rare Word Problem, where infrequent terms lack sufficient data for accurate temporal tracking, and the Scalability Wall, where computational costs explode as vocabularies grow.
To address these, researchers have developed MIST (Multi-output with Importance Sampling Topic model), a framework that leverages word correlations and efficient importance sampling to handle vocabularies of up to 10 million words while improving tracking for rare, specific terms.
Methodology
MIST introduces three primary innovations to overcome the limitations of existing continuous-time models:
- Word Correlation via Multi-output Gaussian Processes (MOGP): Instead of treating word trajectories as independent, MIST embeds words into a latent space. Words proximate in this space (e.g., "neural" and "network") share correlated temporal paths, allowing the model to infer the dynamics of rare words by "borrowing" information from more frequent, related terms.
- Amortized Variational Inference with a Meta-Encoder: The model utilizes a "meta-encoder" that accepts both document content and current topic states. This allows for efficient inference that adapts to "topic drift" without requiring full retraining at each time step.
- Adaptive Softmax Approximation: To maintain scalability, MIST employs self-normalized importance sampling. By sampling a subset of the vocabulary during training rather than calculating probabilities for every word, the computational complexity remains constant relative to vocabulary size.
Experimental Results
Scalability and Efficiency
A core strength of MIST is its ability to scale to massive vocabularies. In synthetic scaling tests, MIST demonstrated near-constant processing time even as the dictionary size reached 10 million words. In contrast, previous state-of-the-art models like DCTM became computationally intractable beyond 1 million words.
Accuracy and Rare Word Tracking
Across seven public datasets—ranging from small collections like State of the Union addresses to large-scale Twitter data—MIST consistently achieved lower perplexity scores than LDA, DETM, and DCTM.
Qualitative analysis further highlights MIST's ability to master rare words. While baseline models often produce noisy or flat trajectories for infrequent technical terms, MIST accurately captures their specific temporal dynamics, closely mirroring actual word counts in the data.
Conclusion and Future Work
MIST represents a significant advancement in dynamic topic modeling, successfully decoupling model performance from vocabulary size while solving the data sparsity issues associated with rare words. Future research may focus on expanding the model's kernel functions to capture anti-correlations between words and exploring non-parametric approaches to allow the number of topics to fluctuate over time. Additionally, further integration of word embeddings could enhance performance on extremely short, sparse documents like single-sentence social media posts.
Summary updated