Abstract
Algorithms such as Latent Dirichlet Allocation (LDA) have achieved significant progress in modeling word-document relationships. These algorithms typically assume each word in the document was generated by a topic sampled independently from a document-specific distribution. This "bag-of-words" assumption ignores the local context and topical flow inherent in human writing. This paper introduces the Hidden Topic Markov Model (HTMM), which relaxes this assumption by modeling topics as a Markov chain. By allowing topics to persist across successive sentences, HTMM achieves superior document modeling and significantly better word sense disambiguation compared to standard LDA.
Introduction
Most traditional topic models rely on the exchangeability of words, assuming that word order does not matter. While computationally efficient, this is linguistically unrealistic. In natural language, topical content tends to be locally consistent; a sentence about a specific concept is likely to be followed by another sentence on the same or a related concept. LDA's failure to capture this local context leads to difficulties in disambiguating word senses (e.g., "support" in a mathematical vs. financial context) and modeling the structural transitions of a document, such as moving from an "Introduction" to a "References" section.
Methodology
The HTMM extends the LDA framework by incorporating a transition mechanism between topics. Instead of sampling a new topic for every word, HTMM introduces a binary transition variable, $\psi_n$, for each word $n$.
As shown in the plate diagrams above, HTMM (c) differs from standard LDA (a) by linking topics $z_n$ in a sequence. If $\psi_n = 0$, the word inherits the topic of the previous word ($z_n = z_{n-1}$). If $\psi_n = 1$, a new topic is sampled from the document's topic distribution $\theta$. To maintain linguistic relevance and computational tractability, the model assumes transitions ($\psi = 1$) can only occur at sentence boundaries, forcing intra-sentence topical consistency. Inference is performed using an Expectation-Maximization (EM) algorithm integrated with the Forward-Backward algorithm to efficiently estimate latent topics.
Experimental Results
The model was evaluated using a collection of 1,740 research papers from the NIPS conferences. The primary quantitative metric was perplexity, which measures the predictive power of the probability model on held-out data.
As illustrated in the figure above, HTMM consistently achieves lower perplexity than LDA. This advantage is particularly pronounced when only a small number of words are observed at the start of a test document, demonstrating HTMM's superior ability to leverage local context for prediction. Qualitatively, the authors demonstrated that HTMM could successfully segment papers into coherent topical blocks and correctly disambiguate words based on their surrounding sequence. For example, the word "support" is assigned to a technical topic in the methods section but switches to a "funding" topic within the acknowledgments.
Conclusion
HTMM represents a significant evolution in topic modeling by relaxing the bag-of-words assumption in favor of a more linguistically grounded Markovian approach. While it relies on heuristics for sentence boundaries and carries higher computational complexity than LDA, its gains in predictive power and interpretability are substantial. Future work may involve integrating more sophisticated structural information, such as syntax or authorship, to further refine the modeling of topical flow.
Summary updated