Museli: A Multi-Source Evidence Integration Approach to Topic Segmentation of Spontaneous Dialogue

Jaime Arguello, Carolyn Penstein Rosé

Abstract

We introduce a novel topic segmentation approach that combines evidence of topic shifts from lexical cohesion with linguistic evidence such as syntactically distinct features of segment initial contributions. Our evaluation demonstrates that this hybrid approach outperforms state-of-the-art algorithms even when applied to loosely structured, spontaneous dialogue.

Introduction

Topic segmentation—the task of automatically identifying where one topic ends and another begins—is a foundational step for information retrieval and summarization. While well-studied for formal texts like news articles, it struggles with spontaneous dialogue due to "lexical sparsity." In a fast-paced conversation, contributions are often terse, and speakers rarely repeat the same keywords across turns. Traditional methods like TextTiling rely heavily on word overlap; when they encounter the short, content-light turns typical of dialogue, they often hallucinate topic boundaries at every single turn.

Museli is a hybrid topic segmentation approach designed specifically for this messy, sparse nature of spontaneous dialogue. By combining traditional lexical cohesion (word repetition) with linguistic features like speaker roles and syntactic markers, it significantly outperforms state-of-the-art algorithms originally designed for formal, expository text.

Methodology

The Museli approach treats segmentation as a supervised binary classification problem: determining if a specific contribution is the start of a NEW_TOPIC or a continuation of the SAME_TOPIC.

Key components of the model include:

  • Hybrid Feature Set: Instead of relying solely on lexical cohesion, Museli integrates linguistic cues (word unigrams, bigrams, and POS bigrams), structural metadata (time difference between turns, contribution length), and speaker roles.
  • Role-Specific Modeling: The model accounts for the agent role (e.g., Tutor vs. Student), recognizing that different roles initiate topics using different linguistic styles. For example, a tutor might use an imperative ("Let's look at..."), while a student might use a question.
  • Classifier: A Naïve Bayes classifier was used, paired with a chi-square test for feature selection to identify the top 1000 most predictive features.

Evaluation Datasets

The researchers evaluated Museli on two distinct dialogue corpora that highlight the challenges of spontaneous speech. As shown in the table below, the Thermo Corpus is particularly challenging due to its extremely short contribution lengths.

Table 1: Statistical comparison of the Olney & Cai and Thermo corpora.
  1. Olney & Cai Corpus: Dialogues with relatively long contributions, averaging 28.63 words per turn.
  2. Thermo Corpus: A corpus of thermodynamics tutoring dialogues with very short, "loosely structured" contributions averaging only 5.12 words per turn.

Key Insights

  • Syntax as a Signal: Topic shifts are often "syntactically marked." High-ranking features for new topics included discourse markers like "now," "then," and "next," as well as imperative bigrams like "lets-do" or "ok-try."
  • The Failure of TextTiling: TextTiling measures term overlap between adjacent regions. Because dialogue turns are so short, many "islands" of zero lexical cohesion exist. TextTiling incorrectly interprets these gaps as topic boundaries.
  • Role Asymmetry: In tutoring scenarios, the tutor is far more likely to drive the topic flow. By training models that "know" who is speaking, Museli captures the specific linguistic habits associated with topic initiation for each role.

Results

Museli outperformed all baselines, including state-of-the-art LSA-based methods (Ortho) and content-oriented models (B&L), by a statistically significant margin ($p < .01$). The primary metric used was $P_k$, where a lower score indicates better performance (measuring the probability of misclassifying two points at a certain distance).

Table 2: Performance comparison (Pk and F-measure) across various segmentation models.

In the Olney & Cai corpus, Museli reduced the probability of error to just 10.51%, a massive improvement over the 27-32% seen in the best baseline models. Even in the challenging Thermo corpus, Museli maintained superior performance compared to traditional lexical methods.

Conclusion and Future Work

The primary drawback of the Museli approach is that it is supervised, requiring hand-labeled training data. Future research will explore bootstrapping techniques—starting with a small amount of labeled data and combining it with high-precision lexical cohesion and discourse markers to automatically expand the training set. Additionally, the authors aim to explore how these models generalize across different domains where speaker roles might vary, such as travel agent and customer interactions.

Summary updated

Abstract

We introduce a novel topic segmentation approach that combines evidence of topic shifts from lexical cohesion with linguistic evidence such as syntactically distinct features of segment initial contributions. Our evaluation demonstrates that this hybrid approach outperforms state-of-the-art algorithms even when applied to loosely structured, spontaneous dialogue.
BibTeX
@inproceedings{arguello2006museli,
  author = {Jaime Arguello and Carolyn Penstein Rosé},
  title = {{Museli: A Multi-Source Evidence Integration Approach to Topic Segmentation of Spontaneous Dialogue}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
  url = {https://openreview.net/forum?id=Sk-jhmZdZH},
}
Figure