More on the normalization of syllable prominence ratings

Christopher Sappok, Denis ArnoldInterspeech 2012

TL;DR

This paper investigates methods to reduce "noise" in subjective syllable prominence ratings, which arise from individual listener biases. By evaluating 12 normalization techniques, the authors demonstrate that specific methods, particularly method 5, can significantly improve the correlation between subjective ratings and acoustic speech features, leading to more reliable prominence data.

What Problem Does This Paper Address?

Subjective ratings of syllable prominence, a key aspect of speech prosody, are crucial for understanding how we perceive speech. However, these ratings are often "noisy" due to individual listener differences in how they interpret and apply the rating scale. This variability, termed inter- and intra-rater variability, can obscure the true relationship between acoustic speech properties and perceived prominence. The authors aim to quantify and reduce this noise by exploring and comparing various normalization methods applied to these subjective ratings.

Methods

The core of this paper is the evaluation of 12 normalization methods for syllable prominence ratings. These methods are systematically categorized based on three factors:

  • Central Tendency (CT): Whether the normalization is based on the mean or median of ratings.
  • Central Tendency Reference Distribution (CTRD): Whether the reference for central tendency is the intra-rating sample (30 syllables) or the intra-listener sample (240 ratings).
  • Dispersion Reference Distribution (DISPRD): Whether dispersion (variability) is normalized, and if so, using intra-rating or intra-listener samples.

The effectiveness of each normalization method is assessed by correlating the normalized rating data with acoustic features (syllable duration, F0, and intensity) using multiple regression. The goodness of fit, measured by adjusted R², serves as the primary metric for evaluating normalization success. The authors also explore different analysis setups, including "ALL" (copying acoustic measures to match rating measures) and "SMALL" (averaging rating measures), and "TOGETHER" (all data combined) versus "SPEAKER-SPECIFIC" analyses.

Datasets

The study utilizes the "Gold"-corpus, a dataset comprising 32 readings of a specific German text ("stanza 3" from Heinrich Heine's "Bimini"). This corpus consists of 960 phonetic syllables, with acoustic and perceptual data collected. The text was read by 8 male German speakers under two conditions: "lyrical" (LYR) and "prose" (PROS) layout, intended to elicit different prominence patterns. Perceptual data was collected from 56 listeners, resulting in a total of 13,440 single rating measures.

Key Insights and Ideas

  • Normalization is Key: Subjective prominence ratings, while informative, are inherently noisy. Normalization techniques are essential for extracting reliable information.
  • Median vs. Mean: The paper suggests that using the median for central tendency normalization might be more robust, as it can manifest base-line shifts as straight zeroes, aligning with hypothesis 4. However, the results indicate mean-based methods (3-8) perform comparably or better than median-based methods (11-16).
  • Dispersion Matters: While initial findings suggested dispersion normalization might be unnecessary, the inclusion of F0 and intensity in the regression models highlights its importance. Method 5, which includes dispersion normalization, consistently outperforms methods without it.
  • Listener vs. Rating Reference: Normalizing with respect to the intra-listener distribution (m=240) for dispersion (as in method 5) is more effective than normalizing with respect to the intra-rating distribution (n=30).
  • Acoustic Features Unlock Prominence: Normalization can reveal correlations with F0 and intensity that are not apparent in raw data, suggesting that these acoustic features play a more significant role in perceived prominence than initially suggested by duration alone.
  • Averaging Simplifies but Loses Information: Averaging raw rating data per syllable (the "SMALL" analysis) leads to higher R² values but sacrifices information about inter-listener variability, which the authors argue is valuable.

Key Results

  • Method 5 Emerges as Superior: Method 5, which normalizes central tendency using the mean and dispersion using the intra-listener mean absolute deviation, consistently yields the highest adjusted R² values when F0 and intensity are included in the regression models.
  • Improved Correlation: Normalization, particularly with method 5, significantly increases the correlation between subjective prominence ratings and acoustic features. For example, for speaker 7, adjusted R² for method 5 normalized data reached .121, compared to .039 for raw data in the "ALL, TOGETHER" analysis.
  • F0 and Intensity Correlations Unveiled: Normalization makes F0-correlations apparent and generally increases INT-correlation above DUR-correlation compared to raw data.
  • Laymen vs. Professionals: Speaker-specific analyses show consistent differences between laymen (speakers 1-4) and professionals (speakers 5-8), with professionals generally achieving higher R² values, indicating more consistent rating behavior.

Limitations and Future Work

  • "SMALL" Analysis Limitation: While averaging raw ratings ("SMALL" analysis) simplifies the data and yields high R² values, the authors note that this approach discards valuable information about inter-listener dispersion. They suggest that if the goal is a single datum per syllable, averaging is sufficient, but if inter-listener dispersion is important, method 5 is recommended.
  • Dispersion Normalization Parameter: The study primarily used mean absolute deviation for dispersion normalization. The authors mention that the consequences of using standard deviation (sd) remain to be investigated, though informal comparisons showed no remarkable differences.
  • Future Work: The paper suggests that further investigation into the role of dispersion and the impact of averaging raw data is warranted. The authors also hint at the potential of normalized prominence ratings for analyzing rhythm phenomena more reliably.

Summary updated

BibTeX
@inproceedings{sappok2012more,
  author = {Christopher Sappok and Denis Arnold},
  title = {{More on the normalization of syllable prominence ratings}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure