Prosody-Based Unsupervised Speech Summarization with Two-Layer Mutually Reinforced Random Walk

Sujay Kumar Jauhar, Yun-Nung Chen, Florian Metze

Introduction

Traditional automatic speech summarization follows a rigid pipeline: transcribing audio via Automatic Speech Recognition (ASR) and then applying text-based summarization algorithms. However, ASR transcripts are often riddled with errors, which propagate through the pipeline and degrade summary quality. Moreover, text-only approaches ignore the rich acoustic channel of human communication—prosody. Speakers use pitch, volume, and pacing to convey emphasis and signal importance. This paper explores whether multi-party meetings can be accurately summarized using only acoustic cues in a completely unsupervised setting, bypassing the noisy lexical layer entirely.

Methodology: Two-Layer Mutually Reinforced Random Walk

The core of the proposed approach is a Two-Layer Mutually Reinforced Random Walk model. This model treats the summarization task as a graph-based ranking problem where importance is determined by the interaction between spoken utterances and their prosodic characteristics.

The Two-Layer Mutually Reinforced Random Walk model architecture, showing the interaction between the Utterance-Layer and the Prosody-Layer.

The pipeline consists of four main stages:

  1. Utterance Chunking & Feature Extraction: Audio is segmented into individual utterances. Prosodic features—including pitch (F0), energy, intensity, speaking rate, and pause counts—are extracted using PRAAT.
  2. Graph Construction: The document is represented as a two-layer graph. The Utterance Layer contains nodes for individual spoken utterances, while the Prosody Layer contains nodes for specific prosodic features. Directed edges connect utterances to the prosodic features they exhibit, with weights determined by the feature values.
  3. Mutually Reinforced Random Walk: Unlike standard PageRank, which operates on a single layer, scores propagate between the two layers. The intuition is that utterances with "important" prosodic features should rank higher, and prosodic features that appear frequently in high-ranking utterances are themselves deemed more significant.
  4. Summarization: The algorithm converges to a closed-form solution (the dominant eigenvector of the transition matrix). The top-scoring utterance nodes are then selected to form the final summary.

Experimental Setup

The model was tested on a corpus of 10 multi-party academic meetings recorded via the SmartNotes system. The dataset involved 6 unique participants and totaled approximately 28 minutes per meeting, comprising 7,123 utterances. Ground truth summaries were established by two human annotators who rated the "noteworthiness" of each utterance on a 1-3 scale; utterances with a score of 3 were used as the gold standard.

Results and Performance

The model was evaluated at 10%, 20%, and 30% extraction ratios using ROUGE metrics and Average Relevance scores.

Comparison of ROUGE-1 and ROUGE-L F-measures between the proposed model and various lexical baselines across different extraction ratios.

The results demonstrate that the prosody-only model is highly competitive:

  • Outperforming Lexical Baselines: At a 30% summary length, the model achieved a ROUGE-1 score of 65.60%, surpassing all baselines, including Latent Topic Entropy (64.97%) and TF-IDF (63.11%).
  • Relevance: The model achieved the highest Average Relevance scores for the 20% (2.422) and 30% (2.411) summaries compared to all baselines.
  • The "Begin" Baseline: The model only trailed the "Begin" baseline (extracting the first N utterances) at the 10% ratio. This is attributed to the structure of academic meetings, which typically start with dense, keyword-heavy agendas that artificially inflate scores for the beginning of the transcript.

Key Insights

  • Prosody as a Proxy for Importance: The model successfully leverages latent speaker states. Features like the number of pauses, minimum/average pitch, and intensity were found to be the most predictive. Frequent pauses often indicate a speaker carefully formulating a thought, which correlates with high relevance.
  • Acoustic vs. Lexical Information: Surprisingly, this purely acoustic model outperformed text-based baselines on longer summaries. This suggests that acoustic delivery carries structural information that word frequencies alone cannot capture.
  • Length Independence: Features related to utterance length (duration, syllable count) were the least predictive, indicating that the model identifies importance based on delivery style rather than just verbosity.

Limitations and Future Work

A current limitation is the reliance on manual transcripts for perfect utterance segmentation. Future work involves integrating the model with automatic speaker diarization and segmentation tools to create a truly end-to-end, raw-audio-to-summary pipeline.

Summary updated

BibTeX
@inproceedings{jauhar2013prosody,
  author = {Sujay Kumar Jauhar and Yun-Nung Chen and Florian Metze},
  title = {{Prosody-Based Unsupervised Speech Summarization with Two-Layer Mutually Reinforced Random Walk}},
  booktitle = {Ijcnlp},
  year = {2013},
}
Figure