Prominence detection in Swedish using syllable correlates

Samer Al Moubayed, Jonas BeskowInterspeech 2010

TL;DR

This paper tackles the challenge of automatically detecting word-level prominence in Swedish speech, a crucial aspect for speech technology applications. They propose a novel linear model that leverages syllable-level acoustic features, outperforming traditional word-level approaches and establishing a new benchmark for Swedish prominence detection.

What Problem Does This Paper Address?

Accurately identifying "prominence" in speech – essentially, which words or syllables stand out perceptually – is vital for applications like speech recognition, synthesis, and comprehension. However, there's a significant mismatch between how humans perceive prominence (often at the word level) and the acoustic cues that signal it (which can be more granular, at the syllable or even phoneme level). This makes it difficult to train models that reliably capture this phenomenon. Specifically for Swedish, existing research highlights that while acoustic correlates like pitch, duration, and intensity exist, their relationship to perceived prominence is complex and not always straightforward, especially when trying to bridge syllable-level acoustics with word-level perception. Furthermore, the scarcity of annotated data, particularly at the syllable level for Swedish, poses a significant hurdle.

Methods

The core of this paper is a linear model designed to estimate prominence. Instead of directly using word-level features, which can be problematic due to the syllable-level realization of acoustic cues, they propose a two-stage approach:

  1. Syllable-Level Feature Extraction: Acoustic features (like fundamental frequency, duration, and loudness) are extracted from the vowel nucleus of each syllable.
  2. Post-Integration of Syllable Prominence: The prominence of each word is then estimated by taking the highest prominence score among its constituent syllables. The weights for the syllable-level features are optimized to minimize the error in predicting the word-level annotations. This "post-integration" step is key to bridging the gap between syllable acoustics and word perception.

They also introduce a linear model for word-level features, where weights are optimized directly to match word-level annotations.

Datasets

The study utilizes a dataset of 200 Swedish sentences read by a professional male actor, originally recorded for speech synthesis. These sentences were phonetically aligned and then annotated for word-level prominence by four speech experts. The annotations were on a three-level scale (Not Prominent, Maybe Prominent, Prominent) and then transformed into numerical values (0, 0.5, 1). The paper notes that this word-level annotation approach was chosen for its practicality and speed compared to syllable-level annotation, despite the theoretical challenge it presents.

Key Insights and Ideas

  • Syllable-Level Features are More Informative: The research strongly suggests that acoustic correlates of prominence are more reliably captured at the syllable level than at the word level, especially for Swedish.
  • Bridging the Gap is Crucial: The proposed "post-integration" method, where the highest syllable prominence determines word prominence, effectively bridges the perceptual gap between syllable acoustics and word-level perception.
  • Linear Models Can Be Powerful: Despite the complexity of prosody, a simple linear model, when applied with carefully chosen features and an appropriate integration strategy, can achieve competitive results.
  • Swedish Prominence Nuances: The paper implicitly highlights the specific characteristics of Swedish prominence, where accent falls and rises are significant, and focus can extend over multiple syllables, making syllable-level analysis particularly relevant.

Key Results

  • The proposed model using syllable-level features achieved the highest accuracy, reaching 86% for prominence detection.
  • This syllable-based approach significantly outperformed a baseline accuracy of 72% and also surpassed models using only word-level features.
  • Among single features, syllable-level "FO range" performed best, exceeding all word-level features and the baseline. Syllable duration also showed improved performance.
  • The best performance (86%) was achieved using a combination of syllable features with optimized weights: Loudness (0.13), Duration (0.52), delta F0 (0.21), F0 (0), and F0 range (0.14).

Limitations and Future Work

  • Single Speaker: The study was conducted on data from a single male speaker. The authors acknowledge that subject-specific feature normalization was not applied, and future work should investigate how the model generalizes to other speakers and genders.
  • Real-time Application: The current model requires pre-calculated parameter distributions. Future research could explore extending the estimation to a real-time scenario by using a moving window approach for parameter calculation.
  • Data Scarcity: While they created a valuable annotated dataset, the paper implicitly points to the ongoing need for more extensive and diverse annotated corpora for prominence research.

Summary updated

BibTeX
@inproceedings{moubayed2010prominence,
  author = {Samer Al Moubayed and Jonas Beskow},
  title = {{Prominence detection in Swedish using syllable correlates}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure