Modeling Acoustic-Prosodic Cues for Word Importance Prediction in Spoken Dialogues

Sushant Kafle, Cissi Ovesdotter Alm, Matt HuenerfauthNAACL 2019

TL;DR

This paper introduces a novel approach to predict word importance in spoken dialogues using acoustic and prosodic features, moving beyond traditional text-based methods. This is crucial for applications like real-time captioning for the deaf and hard-of-hearing, where imperfect ASR output can hinder understanding. The proposed model demonstrates competitive performance, especially when dealing with noisy ASR transcripts.

What Problem Does This Paper Address?

In spoken conversations, not all words carry equal weight in conveying meaning. Identifying these "important" words is vital for tasks like text summarization and, critically, for improving automatic speech recognition (ASR) output, particularly for individuals who are deaf and hard of hearing (DHH). Current ASR evaluation metrics like Word Error Rate (WER) treat all errors equally, failing to capture the nuanced impact of misrecognizing important words. While text-based methods exist for word importance prediction, they struggle with the informal, error-prone nature of conversational transcripts and often require human-generated text, which is not always feasible for real-time applications. This paper addresses the need for a robust word importance prediction system that can operate directly on speech signals, even when ASR output is imperfect.

Methods

The core of this work is a neural architecture designed to model acoustic-prosodic cues for word importance prediction. The approach frames the problem as a sequence labeling task.

  • Acoustic-Prosodic Feature Extraction: The model utilizes a rich set of sub-word acoustic features, including:
  • Pitch (FREQ) and Energy (ENG): Features derived from pitch and energy contours, capturing intonation and emphasis.
  • Spoken-lexical Features (LEX): Word duration, position in utterance, silence duration, syllable count, and articulation rate.
  • Voicing Features (VOC): Spectral tilt (H1-H2) and other measures related to voice quality. These features are extracted using Praat and speaker-normalized, resulting in 60 features per sub-word unit.

  • Sub-word to Word-Level Representation: A bi-directional Gated Recurrent Unit (GRU) network is employed to process sequences of sub-word features, creating a fixed-length vector representation for each word. Spoken-lexical features are then concatenated to this representation.

  • Word Importance Prediction Architecture: A bi-directional Long Short-Term Memory (LSTM) network is used to construct a context-aware representation for each word by processing the sequence of word-level speech features. This contextualized representation is then fed into one of three projection layers for final classification:
  • Softmax Layer: Standard classification for predicting a single label.
  • Relaxed Softmax Layer: Designed for ordinal classification, allowing for multiple label predictions and a "scanning" strategy for final selection.
  • CRF Layer: A linear-chain conditional random field that models dependencies between adjacent word importance labels.

Datasets

The study utilizes a portion of the Switchboard corpus, manually annotated with word importance scores as part of the Word Importance Annotation project. This dataset contains 25,048 utterances from 44 speakers, with word-level timestamps and numeric importance scores ranging from 0 to 1. These scores are categorized into three natural ordinal ranges: Low Importance (LI: [0-0.3)), Medium Importance (MI: [0.3-0.6)), and High Importance (HI: [0.6-1]). The data was split into 80% for training, 10% for validation, and 10% for testing.

For comparison with text-based models, the authors also generated ASR transcripts using Google Cloud Speech API, simulating different Word Error Rate (WER) levels (approximately 25% and 30% with added noise).

Key Insights and Ideas

  • Prosody Matters for Importance: The paper strongly suggests that acoustic and prosodic cues are valuable indicators of word importance, even without textual context. This is a significant departure from purely text-based approaches.
  • Ordinal Classification is Key: The "Relaxed Softmax" projection layer, which explicitly models the ordinal nature of importance (LI < MI < HI), yielded the best performance (lowest RMS error). This highlights that the relative ordering of importance levels is more informative than treating them as independent categories.
  • Robustness to ASR Errors: The speech-based model shows remarkable competitiveness when operating on imperfect ASR transcripts. While text-based models excel on clean text, their performance degrades significantly with ASR errors, making the speech-based approach a valuable alternative or complement.
  • Feature Importance: Speaker normalization (ZNORM) and spoken-lexical features (LEX) were found to be particularly discriminative, with their omission leading to the largest increases in RMS error. Energy-based (ENG) features also played a notable role.

Key Results

  • Ordinal Classification Performance: The LSTM-ORD model achieved the best performance with an RMS score of 68.21, significantly lower than the LSTM-SOFTMAX (74.08) and LSTM-CRF (75.21) models.
  • Comparison with Text-Based Models:
    • On perfect human-generated transcripts, the text-based model outperformed the speech-based model (RMS 54.0 vs. 68.21).
  • However, on ASR transcripts with WER ≈ 25%, the speech-based model (RMS 65.15) was competitive with the text-based model (RMS 72.30).
  • At WER ≈ 30%, the performance was comparable (RMS 68.55 for speech-based vs. 73.6 for text-based).
  • F-score: The speech-based model achieved an F-score of 56 in a 3-class classification task.

Limitations and Future Work

  • Reliance on Human Annotation: The current training and evaluation rely on manually annotated word-level importance scores and word-level timestamps. Automating this annotation process is a key future direction.
  • ASR Output Quality: While the model is robust to ASR errors, the performance still degrades compared to perfect transcripts. The paper acknowledges that the WERs observed (25-30%) are typical for real-world scenarios.
  • Future Work: The authors propose combining both text- and speech-based features for word importance prediction to leverage the strengths of both approaches. This hybrid model could potentially achieve even higher performance and robustness.

Summary updated

BibTeX
@inproceedings{kafle2019modeling,
  author = {Sushant Kafle and Cissi Ovesdotter Alm and Matt Huenerfauth},
  title = {{Modeling Acoustic-Prosodic Cues for Word Importance Prediction in Spoken Dialogues}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure