Abstract
Dialogue Topic Segmentation (DTS) is a critical task for processing long conversations, yet it is hindered by the high cost of human annotation. This paper presents a novel unsupervised framework that trains a BERT-based model to recognize semantic coherence between utterance pairs. By automatically generating a large-scale training corpus from unlabeled dialogues and integrating the resulting coherence scores into the classic TextTiling algorithm, the proposed method achieves state-of-the-art results in both English and Chinese without requiring expensive human-annotated topic boundaries.
Introduction
Dialogue Topic Segmentation (DTS) involves breaking a continuous conversation into smaller, topically consistent segments. It serves as a foundational step for various downstream applications, including dialogue summarization and response prediction. Traditional unsupervised methods have largely relied on surface-level features like lexical overlap, which often fail to capture the fragmented and informal nature of human speech. This research aims to bridge the gap by teaching models to understand the deeper semantic "flow" of a conversation.
Problem Statement
The challenge in DTS is twofold:
- Data Scarcity: Labeling topic shifts is subjective and time-consuming. Consequently, there is very little high-quality labeled data available for training supervised models.
- Shallow Features: Most existing unsupervised methods rely on lexical overlap (counting shared words). However, dialogues are often fragmented; speakers might discuss the same topic using different words or use the same words while switching topics.
Methodology
The researchers propose a three-step pipeline designed to operate without human labels:
- Automatic Data Generation: To train the model, they use a heuristic where adjacent utterances in a real conversation are treated as "coherent" (positive samples). "Incoherent" (negative) samples are created by replacing the second utterance with a random turn from the same dialogue or a different conversation.
- Utterance-Pair Coherence Scoring: A BERT-based model is fine-tuned on this synthetic dataset using a marginal ranking loss. This forces the model to assign a significantly higher coherence score to the "real" adjacent pair than to the "fake" random pair.
- Segmentation Inference: These coherence scores are plugged into the TextTiling algorithm. Instead of word counts, TextTiling uses the BERT-based scores to calculate "depth scores," which measure the sharpness of a coherence "valley." A significant drop in coherence identifies a topic boundary.
Datasets
The approach was evaluated across multiple languages and domains:
- Training: DailyDialog (English) and NaturalConv (Chinese), which are large-scale open-domain conversation datasets.
- Evaluation (English): DialSeg_711 (real-world task-oriented dialogues) and Doc2Dial (synthetic goal-oriented dialogues).
- Evaluation (Chinese): ZYS (real-world banking consultation calls).
Key Insights and Analysis
- Dialogue Flow over Topic Labels: Ablation studies indicated that incorporating "dialogue flows" (the logical sequence of speech acts like Question → Inform) provided a more powerful signal than general topic categories.
- Sharper Boundaries: Visualizing the "depth scores" showed that the BERT-based scorer creates much more distinct "valleys" at topic shifts compared to traditional methods like TF-IDF or raw BERT embeddings.
- Cross-Domain Robustness: Despite being trained on general daily conversations, the model performed exceptionally well on specialized domains like banking and technical support, suggesting that "coherence" is a universal feature of human language.
Key Results
The proposed method outperformed all baselines, including previous state-of-the-art (SOTA) unsupervised models:
- English (DialSeg_711): Achieved an $F_1$ score of 0.776, significantly higher than basic TextTiling (0.608) and BERT-enhanced versions (0.610).
- Chinese (ZYS): Achieved an $F_1$ of 0.521, outperforming the best baseline (0.510).
- Error Reduction: The model showed a marked decrease in $P_k$ and $WD$ (WinDiff) scores, standard metrics where lower values indicate better segmentation accuracy.
Limitations and Future Work
While the results are impressive, the authors identified several areas for future research. The method currently relies on the TextTiling framework for final boundary decisions; future work could explore integrating coherence signals into other structures like GraphSeg. Additionally, incorporating external "commonsense" knowledge or user personas could improve performance in complex, multi-party dialogues. Finally, the slightly smaller performance gains in Chinese suggest that language-specific dialogue structures require further investigation.
Summary updated