Leveraging Summarization for Unsupervised Dialogue Topic Segmentation

Aleksei Artemiev, Daniil Parinov, Alexey Grishanov, Ivan Borisov, Alexey Vasilev, Daniil Muravetskii, Aleksey Rezvykh, Aleksei Goncharov, Andrey V. SavchenkoNAACL 2024 · OpenReview

Abstract

Traditional approaches to dialogue segmentation perform reasonably well on synthetic or written dialogues but suffer when dealing with spoken, noisy dialogs. In addition, such methods require careful tuning of hyperparameters. We propose to leverage a novel approach that is based on dialogue summaries. Experiments on different datasets showed that the new approach outperforms popular state-of-the-art algorithms in unsupervised topic segmentation and requires less setup.

Introduction

Topic segmentation—the task of identifying boundaries where a conversation shifts from one subject to another—is a critical component for downstream applications like meeting summarization, information retrieval, and dialogue systems. While segmentation algorithms have matured for structured written text, they frequently falter when applied to noisy spoken dialogues.

Spoken language is inherently messy, characterized by disfluencies, filler words, and rapid shifts between low-granularity topics. Standard unsupervised methods, such as TextTiling or those relying solely on sentence embeddings, often struggle to distinguish between meaningful topic shifts and transient "chatter." Furthermore, many high-performing models are supervised, requiring large volumes of annotated data that are expensive and time-consuming to produce.

The SumSeg Methodology

To address these challenges, we introduce SumSeg, a five-step unsupervised pipeline that leverages text summarization as a powerful denoising mechanism. The intuition is that summarization models are pre-trained to distill core information, effectively filtering out the noise inherent in raw transcripts.

The SumSeg pipeline: from input text to final segmentation via summarization, embedding, and smoothing.

The SumSeg process consists of the following stages:

  1. Summarization: The dialogue is processed by a pre-trained model (e.g., BART or FLAN-T5) to generate a concise summary.
  2. Simple Sentence Extraction: To ensure each "anchor" represents a single topic, a dependency parser decomposes the summary into "simple sentences," each containing only one main verb.
  3. Embedding: Both the original utterances and the simplified summary sentences are mapped into a shared vector space using sentence embeddings.
  4. Similarity Mapping: A cosine similarity matrix $E$ is constructed, representing the relationship between every utterance in the document and every simple sentence in the summary.
  5. Savitzky–Golay Smoothing: A signal-processing filter is applied to the similarity matrix to remove high-frequency noise, resulting in a smoothed matrix $\hat{E}$.

Finally, a modified TextTiling algorithm identifies segment boundaries based on the local minima in the smoothed similarity scores.

Enhancing Signal through Smoothing

A key innovation in SumSeg is the application of the Savitzky–Golay filter. In raw similarity matrices, momentary mentions of off-topic subjects can create "blips" that lead to over-segmentation. By smoothing the data, the model focuses on sustained topical trends rather than transient fluctuations.

Comparison of the similarity matrix before and after Savitzky–Golay smoothing. The dashed lines indicate predicted topic boundaries.

As shown in the figure above, the smoothing process clarifies the underlying structure of the dialogue, making the transitions between segments significantly more distinct and easier for the TextTiling algorithm to detect accurately.

Experimental Evaluation

SumSeg was evaluated against three diverse benchmarks:

  • SuperDialseg: Document-grounded conversations (9K dialogues).
  • TIAGE: A benchmark focused on human-annotated topic shifts.
  • QMSum: Long-form, multi-domain meeting transcripts (averaging ~9,500 words).

Key Results

SumSeg consistently outperformed established unsupervised baselines like BERTSeg and TopicTiling.

  • Superior Performance on Long Documents: On the challenging QMSum dataset, SumSeg achieved the best scores in both $P_k$ and WindowDiff (WD) metrics. Notably, it outperformed the supervised Bi-H-LSTM model in this setting. This suggests that the summarization-based approach is more robust to the high diversity and length of meeting transcripts than supervised models trained on limited data.
  • Model Agnostic: The method proved effective across various summarization backbones (BART, LED, FLAN-T5), indicating that the architectural approach is more critical than the specific underlying language model.
  • Efficiency: Despite the additional summarization step, inference times remain practical, ranging from under a second to approximately 19 seconds depending on the document length and model size.

Conclusion

SumSeg demonstrates that summarization can serve as an effective "denoising filter" for dialogue topic segmentation. By aligning messy transcripts with clean, simplified summary anchors and applying signal-processing techniques, we can achieve state-of-the-art unsupervised performance. Future work will explore reducing the computational overhead of the summarization step and investigating direct prompting of Large Language Models (LLMs) for even more granular segmentation.

Summary updated

Abstract

Traditional approaches to dialogue segmentation perform reasonably well on synthetic or written dialogues but suffer when dealing with spoken, noisy dialogs. In addition, such methods require careful tuning of hyperparameters. We propose to leverage a novel approach that is based on dialogue summaries. Experiments on different datasets showed that the new approach outperforms popular state-of-the-art algorithms in unsupervised topic segmentation and requires less setup.
BibTeX
@inproceedings{artemiev2024leveraging,
  author = {Aleksei Artemiev and Daniil Parinov and Alexey Grishanov and Ivan Borisov and Alexey Vasilev and Daniil Muravetskii and Aleksey Rezvykh and Aleksei Goncharov and Andrey V. Savchenko},
  title = {{Leveraging Summarization for Unsupervised Dialogue Topic Segmentation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=rE3rXykQgl},
}
Figure