Objective, subjective and linguistic roads to perceptual prominence.how are they compared and why?

Petra Wagner, Fabio Tamburini, Andreas WindmannInterspeech 2012

TL;DR

This paper tackles the challenge of objectively measuring and comparing different aspects of speech prominence (how salient a sound is). They propose a novel tripartite evaluation method that considers acoustic, subjective perceptual, and linguistic expectancy models, offering a more nuanced way to diagnose weaknesses in each. Their findings suggest that current acoustic models may not fully capture human perception, especially in read speech.

What Problem Does This Paper Address?

Perceptual prominence in speech – the feeling that certain syllables or words stand out more than others – is a fundamental aspect of prosody. However, there's a significant lack of consensus on how to measure it. Researchers struggle with:

  1. Subjective Measurement: How to reliably capture what listeners perceive as prominent.
  2. Objective Acoustic Correlates: Identifying the specific acoustic features (like pitch, loudness, duration) that consistently signal prominence.
  3. Relationship to Linguistic Structure: How linguistic factors (like sentence stress, focus, or even just word type) influence perceived prominence.

Traditionally, researchers compare models from these different "layers" (acoustic, subjective, linguistic). However, the authors argue that simply comparing two layers (e.g., acoustic vs. subjective) is often misleading. They point out that mismatches aren't necessarily "mistakes" in one model but can arise from the influence of the third layer – linguistic expectations. They propose a more robust method to evaluate all three layers simultaneously and diagnose where improvements are needed.

Methods

The core contribution is a new tripartite evaluation procedure for prominence models. Instead of pairwise comparisons, they advocate for analyzing mismatches between all three layers:

  • P-model (Perceptual): Subjective ratings of prominence from human annotators.
  • A-model (Acoustic): Prominence values derived from acoustic features.
  • L-model (Linguistic Expectancy): Prominence predicted based on linguistic structure and rules.

They introduce a novel "Mix" similarity measure that combines global and local similarity metrics.

  • Global Similarity (G): Uses the Pearson Correlation Coefficient (PCC) to assess the overall agreement between two profiles.
  • Local Similarity (L): Uses a sliding window approach with Spearman Rank Correlation Coefficient (SRCC) to capture how well peaks and valleys in prominence align locally.
  • Mix Similarity (Mix): A simple average of G and L, providing a single score that balances both global and local agreement.

This "Mix" measure is then used to compare all pairs of models (A vs. P, A vs. L, P vs. L). By analyzing where these pairwise correlations differ, they can infer the influence of the third, "missing" layer. For example, if A and P correlate poorly, but P and L correlate well, it suggests that linguistic expectations (L) are heavily influencing perception (P), and the acoustic model (A) is not capturing this.

Datasets

The study uses a corpus of read German speech produced by three speakers. This corpus contains 285 utterances and is described as "phonetically balanced" and designed to include various prosodic variations (questions, focus, etc.). It has been previously annotated for prominence across the three layers (subjective, objective acoustic, and expectancy-based linguistic).

Key Insights and Ideas

  • Mismatches are Informative: The authors strongly argue that differences between prominence models aren't just errors; they are diagnostic clues about the interplay between acoustic signals, subjective perception, and linguistic expectations.
  • The "Missing Link": Their triangular model highlights how comparing two layers can reveal the influence of the third, unexamined layer. This is crucial for understanding why models might disagree.
  • Beyond Simple Correlation: Standard correlation coefficients (PCC, SRCC) are insufficient because they are too sensitive to the overall distribution of values and can miss local alignment of prominence peaks. The proposed "Mix" measure addresses this by combining global and local similarity.
  • Acoustic Models Lag Behind: The results suggest that current acoustic models for prominence may not fully capture how humans perceive it, especially when compared to linguistic expectations.

Key Results

  • The "Mix" similarity measure provides scores that align better with linguistic intuition for comparing prominence profiles than standard PCC or SRCC, particularly in cases where local alignment is important (as illustrated in Figure 2).
  • In their analysis of the read German corpus, the P-model (subjective) and L-model (linguistic expectancy) generally showed closer agreement with each other than either did with the A-model (acoustic).
  • Specifically, the A-model disagreed with both the L-model and P-model in a significant number of cases (11% of utterances), while the L-model disagreed with the other two in only 3%, and the P-model in only 2%. This strongly indicates that the acoustic model needs further improvement.
  • The authors hypothesize that the read speech style might align well with linguistic expectations, potentially masking the acoustic model's shortcomings. They predict that in more spontaneous speech, the A-model might show better agreement with P-model, while L-model might diverge more.

Limitations and Future Work

  • Speaking Style: The authors acknowledge that the read speech nature of their corpus might influence the results. They predict that spontaneous speech, with its greater stylistic diversity, might yield different patterns of agreement and disagreement between models.
  • Annotator Robustness: They suggest that investigating the behavior of less trained annotators could reveal further problems with the robustness of the subjective (P) model.
  • Further Improvement of Acoustic Models: The primary implication is the need for better acoustic models that can more accurately capture perceptual prominence, potentially by better integrating top-down linguistic information.
  • Spontaneous Speech Analysis: Future work could involve applying this tripartite evaluation to spontaneous speech to see how the relationships between the models change.

Summary updated

BibTeX
@inproceedings{wagner2012objective,
  author = {Petra Wagner and Fabio Tamburini and Andreas Windmann},
  title = {{Objective, subjective and linguistic roads to perceptual prominence.how are they compared and why?}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure