IMS-DTM: Incremental Multi-Scale Dynamic Topic Models

Xilun Chen, K. Selçuk Candan, Maria Luisa SapinoAAAI 2018 · DOI

Introduction

In the rapidly evolving landscape of social media and web data, topics emerge and fade at vastly different rates. Traditional Dynamic Topic Models (DTMs) often fall short by assuming a uniform evolutionary speed across all topics. The Incremental Multi-Scale Dynamic Topic Model (IMS-DTM) addresses this limitation by analyzing data streams at multiple time scales simultaneously, maintaining computational efficiency through a novel incremental Gibbs sampling technique.

The Multi-Scale Dilemma

Standard DTMs require researchers to pre-define a fixed "epoch" or time window. This presents a significant trade-off: large windows capture long-term trends but miss short-lived bursts, while small windows capture fine-grained dynamics at the cost of losing the "big picture" and increasing computational overhead. IMS-DTM posits that topics at various scales co-exist and influence one another; for instance, a long-term research trend provides the necessary context for understanding a short-term algorithmic breakthrough.

Methodology

The IMS-DTM framework introduces two core innovations to handle temporal multi-scale data:

MS-DTM (Multi-Scale Dynamic Topic Model)

This generative model represents the current epoch's topic distribution as a weighted average of past distributions across multiple scales. By employing a Multi-Past Multi-Current (MPMC) scheme, the model observes the "now" through various lenses (e.g., daily, weekly, monthly) concurrently, allowing it to capture both transient spikes and enduring themes.

IMS-DTM (Incremental Inference)

To avoid the high cost of redundant processing in overlapping multi-scale windows, the authors utilize Incremental Multi-Scale Gibbs Sampling. This approach exploits the fact that larger time scales are supersets of smaller ones. By sampling the smallest scale first and then "incrementally" updating those samples for larger scales using the difference ($\Delta$) between them, the model achieves massive computational reuse.

Experimental Evaluation

The model was rigorously tested on three diverse datasets to demonstrate its versatility:

  • NIPS Dataset: Scientific papers representing slow-moving research trends over decades.
  • NYSK (New York v. Strauss-Kahn) Dataset: News articles covering a high-intensity, short-term legal event.
  • Apple Stock Data: Numerical time-series data converted to symbolic "words," illustrating the model's applicability beyond natural language.

Key Results and Insights

The MPMC scheme consistently outperformed single-scale baselines in predictive power, as measured by perplexity. Interestingly, the "sweet spot" for epoch size varied by dataset.

Perplexity comparison across NIPS and NYSK datasets showing the impact of document volume per epoch on model accuracy.

In the NIPS data, larger epochs generally led to higher perplexity, indicating that research trends are better modeled with finer granularity. Conversely, for news data like NYSK, perplexity decreased as the number of documents per epoch increased, suggesting that more data within a window helps stabilize the model for high-intensity events.

Crucially, the incremental approach allows for multi-scale analysis with virtually no execution time penalty compared to single-scale models. Qualitatively, the model successfully maps fine-grained algorithmic terms to broader conceptual categories over time, as shown in the topic evolution timeline.

Qualitative visualization of topic evolution across different time scales, mapping specific terms to broader concepts from past to present.

Conclusion

IMS-DTM provides a robust solution for tracking evolving data streams without sacrificing granularity or speed. By leveraging the inherent overlap in multi-scale data, it offers a computationally efficient way to capture the complex, multi-layered nature of real-world information dynamics. Future research may focus on automated scale selection to further enhance the model's autonomy.

Summary updated

BibTeX
@inproceedings{chen2018ims,
  author = {Xilun Chen and K. Selçuk Candan and Maria Luisa Sapino},
  title = {{IMS-DTM: Incremental Multi-Scale Dynamic Topic Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11988},
}
Figure