Disentanglement of prosodic meaning: Toward a framework for the analysis of nonverbal information in speech

Tirza Biron, Moshe Barboy, Eran Ben-Artzy, Alona Golubchik, Yanir Marmor, Assaf Marron, Smadar Szekely, Yaron Winter, David HarelProceedings of the National Academy of Sciences 2025 · DOI

TL;DR

This paper introduces a novel framework and an automated method for disentangling multiple layers of nonverbal meaning embedded in speech prosody. By treating speech as a hierarchy of "intonation units" (IUs), they develop a modified WHISPER model that can simultaneously detect IU boundaries, prototypes, and emphasis, achieving performance on par with or better than human annotators.

What Problem Does This Paper Address?

Spoken language conveys far more than just the literal meaning of words. Prosody – the rhythm, stress, and intonation of speech – carries crucial nonverbal information such as emphasis, emotional tone, and conversational intent. However, these prosodic messages often co-occur and are difficult to disentangle, making automated analysis challenging. Existing approaches often focus on single prosodic features or rely on form-based annotations rather than functional meaning. This paper aims to bridge this gap by proposing a theoretical framework and a practical method for simultaneously identifying and interpreting these layered, co-occurring prosodic meanings.

Methods

The core of the proposed method is a theoretical framework based on a hierarchy of prosodic messages within "intonation units" (IUs), defined as short speech segments (0.5-2s). They introduce the concept of a "Typed-IU" (TIU), which pairs a basic IU prototype (continuation, conclusion, or request for response) with its specific prosodic variations.

To implement this, they adapt the WHISPER automatic speech recognition (ASR) system. This involves:

  • Fine-tuning: Retraining the WHISPER model to perform a multi-task, multi-label classification for prosodic phenomena.
  • Modified Token Combinations: Adjusting the model's tokenization to better map prosodic information.
  • Ignoring Original Decoded Tokens: This step is crucial for allowing the model to focus on prosodic mapping rather than just speech recognition.
  • Per-Word Prosodic Label Predictor (WPLP): A specific implementation that predicts prosodic tags for each word.

They explore different encoding methods for prosodic labels (Compact vs. Bits) and evaluate the impact of model size.

Datasets

The study utilizes four datasets:

  • This American Life (TAL): A primary dataset containing both spontaneous and read speech, used for training and validation.
  • Interviews Dataset for Prosody Analysis (Interviews): A smaller dataset of spontaneous speech used for validation and zero-shot testing.
  • Aix-MARSEC: A dataset of British English read speech used for comparison and training.
  • Santa Barbara Corpus of Spoken American English (SBC): A large corpus of spontaneous American English speech used for evaluation.

The authors note that existing datasets did not fully align with their functional annotation approach, necessitating in-house tagging for TAL and Interviews.

Key Insights and Ideas

  • Layered Prosody: The paper strongly advocates for viewing prosody not as a single signal, but as a hierarchy of co-occurring messages, each contributing to the overall meaning.
  • Functional Annotation: Emphasizing the importance of annotating prosody based on its communicative function (e.g., "continuation" vs. "conclusion") rather than just its acoustic form.
  • Intonation Units (IUs) as Core Units: IUs are presented as ideal functional units for analyzing prosodic messages due to their natural segmentation and cognitive relevance.
  • Robustness of Multi-Task Learning: The WHISPER model demonstrates that disentangling multiple prosodic messages simultaneously is feasible and doesn't necessarily degrade performance compared to single-task learning.
  • Transfer Learning Efficacy: The fine-tuning approach on a pre-trained ASR model (WHISPER) proves effective, requiring relatively modest amounts of data.

Key Results

  • High Performance: The WHISPER WPLP variant achieved F1 scores for IU boundary detection of up to 0.946 (on TAL), prototype recognition up to 0.789, and emphasis detection up to 0.710.
  • Human-Level Performance: The system's performance is on par with, and often better than, human annotation for similar tasks, as indicated by inter-annotator agreement scores (Cohen's Kappa). For example, TAL-trained V2 model achieved 0.893 CK on IU boundary recognition.
  • Model Size Impact: Larger WHISPER models generally perform better, but the improvement rate diminishes significantly after a certain size.
  • Out-of-Domain Resilience: Models trained on one dataset (e.g., Aix-MARSEC) showed reasonable performance on another (Interviews), indicating some degree of generalization across domains and dialects.
  • Compact Encoding Efficiency: The "Compact encoding" for prosodic labels proved more efficient in terms of execution time.

Limitations and Future Work

  • Incomplete Label Set: The authors acknowledge that their set of prosodic labels is not exhaustive and can be further refined.
  • Dataset Annotation Nuances: The annotation of the TAL dataset, relying more on syntactic cues and resulting in shorter chunks, may have impacted its performance.
  • Improving Feedback Mechanism: The current feedback mechanism in the model could be improved, especially for more complex prosodic interactions.
  • Future Directions:
    • Linguistic Theory: Further defining mutually exclusive prosodic categories and establishing systematic classification trees for TIUs.
  • Distinctive Features: Identifying core prosodic features for IU patterning.
  • Context Formalization: Developing tools to link prosodic patterns with other linguistic components.
  • Formal Annotation System: Creating a standardized "language" for prosodic meaning annotation.
  • Expanding Repertoire: Including more emotions and speaker attitudes, and exploring cross-linguistic prosodic universals.
  • Advanced Context Learning: Developing better multitask learning frameworks for multiple prosodic tasks.

Summary updated

Abstract

We propose a theoretical framework and a cost-effective automated method for the interpretation of prosodic messages (e.g., chunking of information, emphasis, conversation action, emotion). At the core of the proposal is a hierarchy of layered prosodic messages that co-occur within the same intonation unit (0.5 to 2 s long). Motivated by this hierarchy, a procedure for the differential detection of three such co-occurring nonverbal messages is then described. In way of implementation, we produce a variant model of the WHISPER automatic speech recognition system that flags intonation unit boundaries, intonation unit prototypes, and emphases therein. The procedure required us to alter WHISPER's token combinations and significantly adjust its prediction process. The variant model was tested on four datasets that contain spontaneous and read speech, and performs on a par with similar human annotation, and often better, using relatively modest training data. Several insights regarding this implementation, such as model size and encoding methods, are described as well. We believe that the proposed framework, coupled with the results of its application herein, can greatly improve the analysis of speech and language, integrating contextual information and speaker intentions into linguistic descriptions for a large array of purposes with modest means.
BibTeX
@inproceedings{biron2025disentanglement,
  author = {Tirza Biron and Moshe Barboy and Eran Ben-Artzy and Alona Golubchik and Yanir Marmor and Assaf Marron and Smadar Szekely and Yaron Winter and David Harel},
  title = {{Disentanglement of prosodic meaning: Toward a framework for the analysis of nonverbal information in speech}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2025},
  doi = {10.1073/pnas.2500510122},
}
Figure