Regularized-MLLR speaker adaptation for computer-assisted language learning system

Dean Luo, Yu Qiao, Nobuaki Minematsu, Yutaka Yamauchi, Keikichi HiroseInterspeech 2010 · DOI

Abstract

Standard speaker adaptation in Computer-Assisted Language Learning (CALL) often leads to "over-adaptation," where the system incorrectly validates a learner's pronunciation errors by warping the acoustic model to match them. This paper proposes Regularized-MLLR, which constrains adaptation to a mathematical space defined by native speakers. Experimental results demonstrate that this method prevents performance degradation as model complexity increases, significantly improving both pronunciation scoring correlation and error detection accuracy compared to standard Maximum Likelihood Linear Regression (MLLR).

Introduction

In CALL systems, speaker adaptation is essential to account for individual vocal characteristics like pitch and accent. The industry standard, Maximum Likelihood Linear Regression (MLLR), assumes the speaker is attempting "correct" pronunciation. However, for language learners, this assumption fails. If a student consistently mispronounces a phoneme, standard MLLR shifts the model's definition of that phoneme toward the error to maximize data likelihood. This "over-adaptation" effectively moves the goalposts, leading the system to judge errors as correct.

Proposed Method: Regularized-MLLR

To address over-adaptation, the authors introduce a regularization constraint on the transformation matrix. Instead of allowing the matrix to adapt freely to the learner's potentially erroneous speech, it is restricted to a "native manifold":

  1. Teacher Basis Construction: Transformation matrices are first calculated for a group of native English teachers, representing the space of "ideal" vocal variations.
  2. Linear Combination Constraint: The learner’s transformation matrix is forced to be a linear combination of these teacher matrices.
  3. Weight Optimization: The system solves for optimal weights ($\alpha$) that fit the learner's physical voice characteristics (e.g., gender, vocal tract length) while remaining within the space of native-like transformations.

By treating the learner's data only as a means to select the best mixture of "perfect" teacher shifts, the system avoids adapting to phonological errors that do not exist in the teacher space.

Experimental Evaluation

The method was evaluated using two datasets:

  • ERJ (English Read by Japanese Students) Corpus: Used for automatic scoring, containing proficiency labels from experts for 42 learners.
  • Japanese Learners of English Word Corpus: 5,950 utterances annotated with phonemic labels to test error detection.

Results and Discussion

A key finding is the relationship between model complexity (number of regression classes) and performance. In standard MLLR, increasing complexity typically worsens pronunciation evaluation because the model over-fits to errors. Regularized-MLLR remains stable or improves as complexity increases.

Correlation between automated scores and human ratings across different regression class tree sizes.

As shown in the figure above, Regularized-MLLR maintained a high correlation (~0.67) with human raters even as the number of nodes in the regression class tree increased to 64. In contrast, standard MLLR performance peaked early and then crashed as more classes were added, dropping below the baseline "no adaptation" level.

Furthermore, the proposed method significantly improved error detection metrics.

Precision and Recall for error detection across different model complexities.

Regularized-MLLR achieved consistently higher precision and recall than standard MLLR. Notably, while standard MLLR's recall drops as the number of nodes increases, Regularized-MLLR remains stable, identifying more actual errors without sacrificing reliability.

Conclusion and Future Work

Regularized-MLLR effectively decouples physical speaker identity from phonological accuracy in adaptation. Future research will investigate the optimal number and variety of teachers needed for the basis, the minimum adaptation data required from learners, and whether specific phonemes (like liquids or fricatives) require specialized regularization strategies.

Summary updated

BibTeX
@inproceedings{luo2010regularized,
  author = {Dean Luo and Yu Qiao and Nobuaki Minematsu and Yutaka Yamauchi and Keikichi Hirose},
  title = {{Regularized-MLLR speaker adaptation for computer-assisted language learning system}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-231},
}
Figure