Unsupervised Speech Segmentation: A General Approach Using Speech Language Models

Avishai Elmakies, Omri Abend, Yossi AdiarXiv 2025 · DOI · arXiv

Abstract

This paper introduces a novel, unsupervised approach to speech segmentation that focuses on "acoustic-semantic" changes—such as shifts in emotion or speaker gender—rather than low-level phonetic transitions. By leveraging pre-trained Speech Language Models (SLMs) to calculate the mutual information between discrete audio chunks, the authors create a flexible pipeline that outperforms traditional baselines without requiring task-specific training data.

Introduction

Traditionally, speech segmentation is categorized into low-level spectral segmentation (e.g., phoneme or word boundaries) and specialized supervised tasks like speaker or emotion diarization. However, a gap exists for a general, unsupervised approach capable of segmenting audio based on higher-level acoustic-semantic styles—features that do not translate well into text, such as sudden emotional shifts or gender changes. Instead of training bespoke supervised models for every possible style change, this research explores using the inherent distributional knowledge of pre-trained Speech Language Models (SLMs) to detect these boundaries automatically.

Methodology

The proposed pipeline is inspired by text-based NLP segmentation techniques, adapted for continuous audio through three primary stages:

  1. Sentencer: The raw audio is initially divided into equal-sized "acoustic-sentences." Empirical testing indicated that 0.5-second chunks provide the most effective granularity for subsequent analysis.
  2. Scorer: Audio is quantized into discrete tokens using a pre-trained HuBERT model. A pre-trained SLM (TWIST 350M) then calculates the Point-Wise Mutual Information (PMI) between consecutive acoustic-sentences. Low or negative PMI suggests that two chunks are likely independent, signaling a probable segmentation boundary.
  3. Span Selector: An algorithm determines final boundaries based on PMI scores using one of three methods: Constant (fixed number of segments $k$), Adaptive (dynamic based on audio length), or Threshold (boundary placed where PMI drops below a value $t$).
The proposed unsupervised speech segmentation pipeline, illustrating the flow from raw audio to final segmented spans.

Experimental Framework

To evaluate this new framing of the segmentation problem, the authors developed synthetic benchmarks by concatenating clips from two emotional speech databases: EmoV-DB (Emotional Voices Database) and IEMOCAP (Interactive Emotional Dyadic Motion Capture). Two experimental setups were designed:

  • Change of Emotion: Concatenating clips of different emotions from the same speaker.
  • Change of Gender: Alternating clips between male and female speakers while maintaining a constant emotion.

Key Results and Analysis

The PMI-based method was evaluated against an Equal Length (EL) baseline, a Contrastive Learning Scorer (CLS), and state-of-the-art supervised Diarization Methods (Speaker Diarization [SD] and Emotion Diarization [ED]).

  • Superior Boundary Detection: Using the "Adaptive" span selector, the proposed method achieved a Purity Coverage F1 (PC-F1) of 72.0 on the EmoV-DB emotion change task, significantly outperforming the Contrastive Learning baseline (50.0).
  • Reduced Over-segmentation: While supervised ED models suffered from extreme over-segmentation (yielding R-Values as low as -95.6), the unsupervised SLM approach maintained strong, positive R-Values (e.g., 41.4), indicating cleaner and more logical boundaries.
  • Parameter Sensitivity: The choice of initial chunk size in the Sentencer stage is critical. As shown in the performance metrics, smaller acoustic-sentence sizes (0.5s) consistently yielded higher metric values across both emotion (E) and gender (G) tasks.
Impact of acoustic-sentence size on segmentation performance metrics (PR-F1 and R-Val) for emotion (E) and gender (G) change tasks.

Discussion and Future Work

Key Insights

Pre-trained SLMs inherently capture high-level acoustic-semantic information in their discrete token representations, allowing them to recognize style changes without explicit supervision. Furthermore, classic NLP metrics like PMI translate effectively to these discrete speech tokens, providing a robust, zero-shot alternative to specialized supervised models that often fail when applied out-of-distribution.

Limitations and Future Directions

The current approach is computationally intensive, requiring significant GPU resources for inference. Additionally, the naive 0.5s initial chunking acts as a precision bottleneck. Future work will explore replacing fixed-length chunks with variable-length acoustic-sentences (e.g., via unsupervised phoneme detection) and developing SLMs that explicitly incorporate prosodic features like pitch and duration to enhance segmentation accuracy.

Summary updated

BibTeX
@misc{elmakies2025unsupervised,
  author = {Avishai Elmakies and Omri Abend and Yossi Adi},
  title = {{Unsupervised Speech Segmentation: A General Approach Using Speech Language Models}},
  year = {2025},
  eprint = {2501.03711},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.03711},
}
Figure