Traditional topic models like Latent Dirichlet Allocation (LDA) treat documents as a "bag of words," assuming that the order of sentences or utterances doesn't matter. However, human discourse is inherently linear and structured. Whether it is a news report, a political manifesto, or a casual conversation, topics do not appear at random; they flow logically from one to the next.
Researchers have addressed this by moving toward sequential topic models that treat a document or a corpus as a path through a series of latent states. By applying Markovian assumptions to topic distributions, these models capture the "narrative skeleton" of text, allowing machines to understand not just what is being discussed, but how the discussion evolves.
The Intuition: Topical Inertia and Flow
The core insight behind this research direction is that topics exhibit inertia. If a speaker is currently discussing "economic policy," the probability that the very next sentence is also about "economic policy" is much higher than a sudden jump to "marine biology."
Cäcilia Zirn and colleagues demonstrated this in political manifestos, showing that topics tend to persist across multiple sentences. By using Markov Logic Networks to account for this persistence, they significantly improved classification accuracy over models that looked at sentences in isolation.
This flow is often modeled using a transition matrix, which defines the probability of moving from topic $A$ to topic $B$. Hongning Wang and ChengXiang Zhai introduced the Structural Topic Model (strTM), which explicitly learns these transitions. This allows the model to resolve ambiguity: if a sentence contains words that could belong to two different topics, the model uses the topic of the preceding sentence as a structural prior to "nudge" the classification toward the most logically consistent choice.
Modeling Narrative Structure with Hidden Markov Models
In specific domains, the flow of information follows a predictable "script." For example, an earthquake report usually starts with the location and magnitude before moving to casualties and then aid efforts.
Regina Barzilay and Lillian Lee pioneered probabilistic content models using Hidden Markov Models (HMMs) to capture this domain-specific flow. In their framework:
- Hidden States represent distinct topics (e.g., "Casualties").
- Transitions capture the typical narrative progression.
- Emission Distributions generate the specific words associated with that topic.
This approach has been extended to creative and functional tasks. Kento Watanabe used HMMs to ensure thematic consistency in AI-generated lyrics, modeling the shift from descriptive verses to emotional choruses. Similarly, Renxian Zhang used "textual aspects" as an intermediary semantic layer to ensure that multi-document summaries follow a human-like logical order.
Segmentation and Discovery of Boundaries
If topics are sequential, then a change in topic signals a structural boundary. Matthew Purver and Joshua B. Tenenbaum applied this to multi-party discourse (like meetings) by integrating a binary switching variable into the generative process. At each utterance, the model decides whether to inherit the previous topic or sample a new one. This allows the model to simultaneously identify what is being discussed and where the conversation shifts.
Recent work has scaled this logic to ultra-long texts. Yan Yan combined Markov chain principles with Large Language Models (LLMs) to generate outlines for long Chinese novels, using the historical progression of the story to prevent inconsistent boundary detection.
Temporal Evolution and External Shocks
The Markovian assumption also applies to how topics evolve over time across a whole corpus. In Dynamic Topic Models, the state of a topic at time $t$ is conditioned on its state at $t-1$.
- Public Health: Simon Kocbek and Wray L. Buntine used this to track seasonal spikes in respiratory issues within noisy emergency room notes.
- Social Media: Yulan He developed the dynamic Joint Sentiment-Topic (dJST) model to track how both topics and public sentiment shift across sequential epochs of social media data.
- External Influences: Baiyang Wang and Diego Klabjan refined this by distinguishing between endogenous evolution (natural thematic drift) and exogenous processes (sudden shifts caused by real-world events like economic shocks).
Why It Works: Contextual Constraints
The success of these models stems from their ability to relax the independence assumption. By acknowledging that the "latent state" of a word or sentence is dependent on its neighbors, these models capture the local context and semantic continuity that bag-of-words models ignore.
As Jun Zhu demonstrated with Conditional Topic Random Fields, modeling dependencies between the topic assignments of adjacent words allows a system to capture nuances like negation and local sentiment. By treating text as a structured sequence rather than a random collection, these models align more closely with the way humans actually produce and consume information.