The Automatic Assessment of Non-native Prosody: Combining Classical Prosodic Analysis with Acoustic Modelling

Florian Hönig, Tobias Bocklet, Korbinian Riedhammer, Anton Batliner, Elmar NöthInterspeech 2012 · DOI

Introduction

Mastering a second language (L2) involves more than just correct grammar and vocabulary; it requires the acquisition of the language's "music"—its prosody. Prosody encompasses rhythm, stress, and intonation (melody). Deviations in prosody can significantly impede a speaker's intelligibility and contribute to a perceived foreign accent, even when segmental features like vowels and consonants are accurate.

While automated systems for pronunciation training have improved, providing nuanced, continuous feedback on prosodic quality remains a challenge. Traditional "knowledge-heavy" approaches rely on precise temporal segmentation, while newer "agnostic" acoustic models offer a different perspective. This paper explores the benefits of fusing these two methodologies to create a more robust automatic assessment tool.

Methodology: Fusing Knowledge and Agnosticism

The researchers investigated three distinct feature sets to predict human ratings of L2 speech using Support Vector Regression (SVR):

  1. Prosodic Features (pros): This set includes 742 features derived from fundamental frequency (F0), energy, duration, and pauses. This approach is "knowledge-heavy" as it requires forced alignment—knowing the exact timing of words and syllables—to calculate metrics like syllable duration or pitch contours.
  2. GMM-UBM Features (gmm): Borrowed from speaker recognition technology, this method uses a Gaussian Mixture Model (GMM) to establish a Universal Background Model (UBM) of native speech. It measures how much a non-native speaker's spectral characteristics deviate from this native norm. Crucially, this is a "black-box" approach that does not require a transcript or segmentation.
  3. openSMILE Features (smile): This toolkit extracts 4,368 general-purpose acoustic features (e.g., loudness, spectral jitter). It treats the audio as a "flat" block of sound without linguistic context.

The study employed late fusion, where the outputs of individual SVR models trained on these feature sets were combined to produce a final, optimized score.

Data and Human Ground Truth

The system was evaluated using a combined dataset from the C-AuDiT and ISLE databases, featuring 58 L2 speakers from diverse linguistic backgrounds (German, French, Spanish, Italian, and Hindi) and 11 native American English speakers.

To establish a ground truth, 62 native English listeners rated five prosodically challenging sentences on a 5-point Likert scale across four dimensions: Intelligibility, Accent, Melody, and Rhythm. Analysis of these human ratings revealed high inter-correlations, particularly between melody and rhythm.

Correlation between human ratings for Rhythm and Melody (r=0.950).

As shown in the figure above, the perception of rhythm and melody is nearly inseparable for human listeners, justifying their combined treatment in prosodic assessment systems.

Key Results

The performance was measured by the Pearson correlation ($r$) between the automated scores and the averaged human ratings.

  • Rhythm and Melody: The classical prosodic model performed well ($r = 0.793$ for rhythm, $r = 0.735$ for melody). However, fusing these with GMM-UBM features yielded significant improvements, reaching $r = 0.822$ for rhythm and $r = 0.771$ for melody.
  • Intelligibility and Accent: The fusion approach also proved superior here ($r = 0.683$ and $0.693$ respectively). Interestingly, the GMM-UBM features, despite being agnostic to prosody, provided critical complementary information that classical features lacked.
  • The openSMILE Baseline: The general-purpose openSMILE features performed the worst, indicating that "blind" features are insufficient for prosody assessment without some form of structural or linguistic context.
Correlation between human ratings for Accent and Intelligibility (r=0.828).

Discussion and Conclusion

The primary takeaway is that "agnostic" acoustic models capture subtle spectral deviations that traditional prosodic analysis misses. By combining these "alien" features with knowledge-heavy prosodic metrics, systems can more accurately mirror human perception.

A current limitation is the reliance on fixed sentences. Future research will focus on developing sentence-independent models and more sophisticated ways to integrate general acoustic features into the temporal structure of speech. This work represents a significant step toward more effective and nuanced Computer-Assisted Pronunciation Training (CAPT) tools.

Summary updated

BibTeX
@inproceedings{honig2012automatic,
  author = {Florian Hönig and Tobias Bocklet and Korbinian Riedhammer and Anton Batliner and Elmar Nöth},
  title = {{The Automatic Assessment of Non-native Prosody: Combining Classical Prosodic Analysis with Acoustic Modelling}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-250},
}

Loading related content...

Figure