Obtaining prominence judgments from naive listeners.influence of rating scales linguistic levels and normalisation

Denis Arnold, Petra Wagner, Bernd MöbiusInterspeech 2012

TL;DR

This paper investigates how to best capture human judgments of speech prominence, finding that using more granular rating scales and rating at the word level yields more reliable and acoustically correlated results. They also demonstrate that normalizing these ratings can further improve their robustness and alignment with linguistic expectations.

What Problem Does This Paper Address?

Understanding and quantifying speech prominence is crucial for many areas of speech processing, including prosody modeling, speech synthesis, and automatic speech recognition. However, there's no single, universally agreed-upon method for eliciting and analyzing prominence judgments from human listeners. This leads to challenges in comparing results across different studies, as variations in rating scales, the linguistic units being rated (words vs. syllables), and data normalization can significantly influence findings. This paper aims to systematically evaluate these factors to provide a more robust methodology for obtaining reliable prominence judgments.

Methods

The authors leverage existing experimental data from two previous studies ([17] and [18]) that collected prominence ratings from naive listeners. They compare different rating scales (binary, 4-point, 11-point, 31-point, and continuous) and analyze prominence judgments at both the word and syllable levels. A key methodological contribution is the application and evaluation of a z-transformation for normalizing prominence ratings. This normalization technique aims to standardize ratings across listeners and conditions, making them more comparable and potentially revealing underlying patterns more clearly.

Datasets

The paper utilizes data from two previously conducted experiments ([17] and [18]). These datasets consist of prominence ratings collected from naive German listeners on sentences. No new datasets were introduced in this paper; rather, existing data was re-analyzed and evaluated through the lens of different methodological choices.

Key Insights and Ideas

  • Multilevel Scales are Better: Contrary to some assumptions, using scales with more levels (like 11-point or 31-point) provides richer insights, allowing researchers to distinguish between perceived prominence and listener confidence, and generally leads to more reliable results.
  • Word Level is King (for Naïve Listeners): Naïve listeners appear to be more adept at rating prominence at the word level than at the syllable level. This leads to stronger correlations with acoustic features and is generally easier for listeners.
  • Normalization is Crucial: Normalizing prominence ratings (using z-transformation) significantly improves the correlation between these subjective judgments and objective acoustic correlates. It also helps align prominence patterns with linguistic expectations, correcting for potential listener biases or variations.
  • Priming Effects are Robust (with the right tools): The study reinforces that priming listeners with prominence patterns can be effective. Crucially, normalization helps to replicate these priming effects even with scales that might otherwise obscure them.

Key Results

  • Correlations between acoustic features (like duration, maximum f0, and intensity) and prominence ratings were generally higher for word-level prominence compared to syllable-level prominence.
  • The use of multilevel scales (specifically 11-point and 31-point) was found to be more advantageous than binary or 4-point scales for capturing prominence patterns and replicating priming effects.
  • Normalization using z-transformation improved the correlation between acoustic correlates and prominence ratings in both word and syllable data sets.
  • After normalization, prominence ratings were more in line with linguistic expectations, as illustrated by an example sentence where the expected prominence of the final syllable was better reflected.
  • Priming effects, which were only replicated with the 31-point scale on raw data, could be replicated with both the 11-point and 31-point scales after normalization.

Limitations and Future Work

  • Listener Confidence vs. Prominence: While multilevel scales allow for observing listener confidence, the paper notes that high confidence doesn't necessarily equate to extreme prominence ratings, and vice-versa. Further investigation into the interplay between confidence and actual prominence judgments is suggested.
  • Comparing Across Levels: The authors emphasize the need for caution when comparing prominence results across studies that use different linguistic levels (word vs. syllable) for annotation, as their findings show significant differences.
  • Generalizability: While the study uses German data, future work could explore whether these findings generalize to other languages.
  • Further Normalization Exploration: The paper focuses on z-transformation. Exploring other normalization techniques could be a fruitful avenue for future research.

Summary updated

BibTeX
@inproceedings{arnold2012obtaining,
  author = {Denis Arnold and Petra Wagner and Bernd Möbius},
  title = {{Obtaining prominence judgments from naive listeners.influence of rating scales linguistic levels and normalisation}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure