Integration of multilayer regression analysis with structure-based pronunciation assessment

Masayuki Suzuki, Yu Qiao, Nobuaki Minematsu, Keikichi HiroseInterspeech 2010 · DOI

Abstract

Automatic pronunciation assessment faces significant challenges due to acoustic variations unrelated to language proficiency, such as age, gender, and individual vocal tract characteristics. This paper proposes a hierarchical approach that integrates invariant speech structures with multilayer regression analysis to achieve robust scoring. By focusing on the relationships between phonemes rather than absolute acoustic patterns, the system achieves a 0.75 correlation with human graders, nearly matching human-to-human consistency.

Introduction

Traditional automatic pronunciation assessment methods, such as the Goodness of Pronunciation (GOP) score, often struggle when a student's voice deviates significantly from the native speaker models used for training. While speech structure models have been proposed to mitigate this by focusing on relative phonemic distances, they often suffer from high dimensionality that can impede accuracy. This study addresses these limitations by introducing a multilayer regression framework to manage structural complexity and a hybrid integration with traditional acoustic features.

Methodology

Invariant Speech Structure

The core of the proposed method is the construction of a speaker-invariant speech structure. Instead of analyzing absolute spectral features, the system calculates the f-divergence (specifically the Bhattacharyya distance) between different phoneme distributions. This process generates a distance matrix representing the relative "spacing" between sounds.

Structural invariance across different acoustic spaces.

As illustrated in the figure above, while the absolute positions of phoneme distributions ($p_i$ and $q_i$) may shift between different acoustic spaces (representing different speakers), the underlying structure—the web of relative distances—remains stable. This invariance makes the model naturally robust to speaker-specific traits.

Multilayer Regression Analysis

To overcome the "curse of dimensionality" associated with large distance matrices, a three-layered regression strategy is employed:

  1. Layer 1 (Phoneme Level): Analyzes individual rows of the distance matrix to estimate proficiency for specific phonemes.
  2. Layer 2 (Speaker Level): Aggregates these phoneme-level estimates to predict a comprehensive proficiency score for the student.
  3. Layer 3 (Integration): Combines multiple "teacher" models and diverse acoustic features to refine the final prediction.

Hybrid Integration

The structural analysis is finally integrated with traditional GOP scores. This hybrid approach leverages the speaker-invariant "contrastive" strengths of speech structures (particularly effective for vowels) and the "absolute" acoustic strengths of GOP (effective for unvoiced consonants).

Experimental Evaluation

The system was evaluated using the English Read by Japanese (ERJ) database, comprising recordings from 200 Japanese university students and 20 native speakers of General American English. Ground truth was established using proficiency scores from five native English teachers.

Results

The proposed three-layered regression model, when combined with GOP, achieved a 0.75 correlation with human scores. This performance is highly significant, as the average correlation between two human experts in the same dataset was 0.77.

Furthermore, the system demonstrated exceptional robustness. In tests involving "warped" speech to simulate extreme vocal tract variations, the structure-based method maintained stable performance while the standard GOP method's accuracy declined sharply.

Conclusion

By shifting the focus from absolute acoustic values to the structural relationships between sounds, this research provides a more human-like and robust framework for pronunciation assessment. Future work will focus on utilizing phoneme-level human labels to improve diagnostic feedback, allowing the system to provide specific instructions to students on which sounds require improvement.

Summary updated

BibTeX
@inproceedings{suzuki2010integration,
  author = {Masayuki Suzuki and Yu Qiao and Nobuaki Minematsu and Keikichi Hirose},
  title = {{Integration of multilayer regression analysis with structure-based pronunciation assessment}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-229},
}
Figure