Word prominence detection using robust yet simple prosodic features

Taniya Mishra, Vivek Rangarajan Sridhar, Alistair ConkieInterspeech 2012

TL;DR

This paper introduces a novel set of prosodic features for automatically detecting word prominence, a crucial element for understanding spoken language. These new features, which capture the shape and magnitude of pitch contours without relying on explicit peak detection, significantly outperform traditional prosodic features and even rival lexical/syntactic approaches on spontaneous speech.

What Problem Does This Paper Address?

In spoken language, speakers use "prominence" to highlight certain words, conveying emphasis, new information, or emotional tone. Automatically identifying these prominent words is vital for downstream Natural Language Understanding (NLU) tasks. However, accurately detecting prominence is challenging because the acoustic cues (like pitch, energy, and duration) are often noisy, especially when derived from automatic speech recognition. Previous work often relied on aggregating these acoustic cues (e.g., mean, max, variance) or using lexical/syntactic information. While effective, aggregation loses detailed information about pitch contour shape, and lexical features can be unreliable due to speech recognition errors or lack of generalizability. This paper aims to bridge this gap by developing prosodic features that are both robust to noise and capture more nuanced acoustic information than simple aggregations.

Methods

The authors propose a new set of prosodic features designed to capture the dynamics of the fundamental frequency (F0) contour, energy, and duration within a word. Unlike prior methods that explicitly identify F0 peaks or valleys (which can be difficult in noisy or complex contours), these new features are computed directly from the smoothed F0 and energy contours.

Key novel features include:

  • Area under the F0 curve (AFC): Integrates F0 and duration to capture overall prominence.
  • Energy-F0-Integral (EFI): Similar to AFC but also incorporates energy, acknowledging its role in prominence.
  • Voiced-to-unvoiced ratio (VUR): Acts as a reliability measure for the AFC and EFI features, indicating how much of the F0 contour is based on actual voicing versus interpolation.
  • F0 curve shape (SHP): Uses isotonic regression to estimate the likelihood of different contour shapes (rising, falling, peak, valley) within a word and its neighbors, providing a richer representation than simple statistics.
  • F0 peak/valley amplitude and location (FAMP and FLOC): Estimates peak/valley characteristics if they are detected by the isotonic regression, but in a more robust way than traditional methods.

These new features are combined with standard prosodic features (like mean, median, variance of F0 and energy) and lexico-syntactic features (word identity, part-of-speech tags, word type, syllable count, break tags) for comparison. The models are trained using ensemble methods: Random Forest and AdaBoost, chosen for their efficiency with high-dimensional, unbalanced datasets and their ability to rank feature importance.

Datasets

The study uses a subset of the Switchboard corpus, which includes manually corrected word segmentations and hand-marked prominence labels. This dataset consists of approximately 67,000 word instances, with about one-third marked as prominent. The prominence labels were transduced to a binary (prominent/non-prominent) format at the word level. This dataset has been previously used for prominence detection, providing a benchmark for comparison.

Key Insights and Ideas

  • Shape Matters More Than Simple Aggregation: The paper strongly suggests that capturing the shape and dynamics of prosodic contours (like F0) is more informative for prominence detection than simply using aggregate statistics (mean, max, variance).
  • Robustness to Noise is Key: The proposed features are designed to be robust to the inherent difficulties of identifying precise acoustic landmarks (like F0 peaks) in real-world speech, making them more reliable for downstream applications.
  • Prosodic Features Can Outperform Lexical/Syntactic: In spontaneous speech, the novel prosodic features, when used alone, achieve better accuracy than lexico-syntactic features, highlighting their power and generalizability.
  • Feature Importance Reveals Value: The feature importance analysis (Gini index) clearly shows that the newly developed prosodic features are among the most discriminative, with word duration being the single most important feature overall.

Key Results

  • Superior Performance of New Features: The new prosodic features, when used in a model, achieved an accuracy of 77.2%, significantly outperforming the standard aggregate prosodic features (74.1%).
  • Combined Power: A model using both the new prosodic features and lexico-syntactic features (excluding word identity) achieved the highest accuracy of 81.5%.
  • Prosodic vs. Lexico-Syntactic: Under comparable conditions (without break tags), the prosodic-feature-based model (77.2%) was more accurate than the lexico-syntactic model (75.0%).
  • Feature Importance: Duration was the most important feature. Among the top 10 features, six were the newly developed prosodic features, indicating their high discriminative power.

Limitations and Future Work

  • Word Identity: The authors acknowledge that word identity is a strong feature but exclude it from their final model because it does not generalize to unseen data. They suggest that their novel prosodic features offer a more generalizable alternative.
  • DLH Feature: The "Average difference between low and high frequency components (DLH)" feature showed little to no discriminative power and was excluded from the best-performing model.
  • Future Work: The paper implicitly suggests that further exploration of feature combinations and potentially more sophisticated modeling techniques could lead to even greater improvements. The robustness and effectiveness of the proposed prosodic features open avenues for their application in various speech processing tasks beyond prominence detection.

Summary updated

BibTeX
@inproceedings{mishra2012word,
  author = {Taniya Mishra and Vivek Rangarajan Sridhar and Alistair Conkie},
  title = {{Word prominence detection using robust yet simple prosodic features}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure