Abstract
This paper describes a novel Bayesian approach to unsupervised topic segmentation. Unsupervised systems for this task are driven by lexical cohesion: the tendency of well-formed segments to induce a compact and consistent lexical distribution. We show that lexical cohesion can be placed in a Bayesian context by modeling the words in each topic segment as draws from a multinomial language model associated with the segment; maximizing the observation likelihood in such a model yields a lexically-cohesive segmentation. This contrasts with previous approaches, which relied on hand-crafted cohesion metrics. The Bayesian framework provides a principled way to incorporate additional features such as cue phrases, a powerful indicator of discourse structure that has not been previously used in unsupervised segmentation systems. Our model yields consistent improvements over an array of state-of-the-art systems on both text and speech datasets.
Introduction
Topic segmentation—the task of partitioning a document or transcript into topically coherent chunks—is a fundamental challenge in natural language processing. Traditionally, unsupervised systems for this task have relied on the principle of lexical cohesion: the observation that a well-formed segment tends to use a consistent and compact set of words.
However, previous methods often relied on hand-crafted heuristics, such as weighted cosine similarity or lexical chains, to quantify cohesion. These metrics are difficult to generalize across domains and hard to integrate with other structural indicators, such as discourse markers or "cue phrases," without resorting to supervised learning. This paper proposes a generative Bayesian model that transforms lexical cohesion from a heuristic into a formal mathematical consequence of probability conservation.
Methodology
The authors propose a generative model where words in each segment are treated as draws from a segment-specific multinomial language model.
Lexical Cohesion as Probability
In this framework, lexical cohesion is not a manual metric but a natural byproduct of the model. Because probability mass must be conserved, a model that concentrates its probability on a small subset of words (a cohesive segment) yields a higher likelihood than one that spreads it thinly across a diverse vocabulary. The authors explore two inference methods:
- Point Estimates: Explicitly estimating the language models for each segment.
- Marginalization: Using the Dirichlet Compound Multinomial (DCM) distribution to integrate out the language models, allowing the inference to focus solely on segment boundaries.
Incorporating Cue Phrases
A significant advantage of this Bayesian framework is its ability to incorporate non-lexical features like cue phrases (e.g., "now," "anyway," "however"). The authors introduce a second language model specifically for words appearing at segment boundaries. This allows the system to learn, in an unsupervised manner, that certain words are strong indicators of a new topic, even if they lack topical content themselves.
Algorithms
For the base model, Dynamic Programming is used to find the exact maximum-likelihood segmentation. When cue phrases are added—coupling segments across the document—the authors employ Metropolis-Hastings sampling for approximate inference.
Evaluation and Results
The model was evaluated on two distinct domains: the ICSI Meeting Corpus (multi-party speech transcripts) and a Medical Textbook Corpus (227 chapters of formal written language). Performance was measured using $P_k$ and WindowDiff (WD), where lower scores indicate better performance.
As shown in the results table, the Bayesian model (BAYESSEG) consistently outperformed state-of-the-art baselines like U&I, MCS, and LCSEG.
- Medical Text:
BAYESSEGachieved a $P_k$ of 0.339, significantly better than the best baseline (MCS at 0.368). - Meetings: The full model incorporating cue phrases (
BAYESSEG-CUE-PROP) achieved the best performance with a $P_k$ of 0.258, compared to the best baseline (U&I at 0.297).
An interesting finding was the domain-specific utility of cue phrases. While they significantly improved performance in the informal, messy setting of meetings, they provided negligible benefit for the highly structured medical textbook. This demonstrates the model's robustness in ignoring non-informative features for a given domain.
Discussion
The authors demonstrate that their Bayesian objective is mathematically equivalent to minimizing the weighted sum of entropies for each segment in specific cases. This provides a theoretical bridge between generative modeling and information theory. Furthermore, they show that previous successful models, such as Utiyama and Isahara (2001), can be derived as special cases of this Bayesian framework using Laplacian smoothing.
Conclusion
By moving from hand-crafted heuristics to a formal probabilistic framework, this research provides a principled way to perform unsupervised topic segmentation. The model not only improves performance on standard benchmarks but also allows for the seamless integration of diverse features like cue phrases. Future work may extend this approach to hierarchical segmentation and the inclusion of acoustic features like prosody for speech data.
Summary updated