Spoken English assessment system for non-native speakers using acoustic and prosodic features

Qin Shi, Kun Li, ShiLei Zhang, Stephen M. Chu, Ji Xiao, ZhiJian OuInterspeech 2010

TL;DR

This paper introduces an automatic system for assessing spoken English proficiency in non-native speakers, focusing on acoustic and prosodic features beyond simple pronunciation. It aims to provide immediate, detailed feedback to learners, leveraging ASR, specialized feature extraction for word stress and rhythm, and a summarization stage for overall scoring.

What Problem Does This Paper Address?

Learning a foreign language, particularly English, often suffers from a lack of real-time, targeted feedback, especially for pronunciation and fluency. Traditional classroom settings can't always provide this individualized attention. While automatic speech recognition (ASR) systems can offer some insights, they often fall short in providing nuanced feedback on specific pronunciation errors, word stress, and the natural flow (rhythm and fluency) of speech. This paper addresses this gap by developing a system that goes beyond basic acoustic analysis to capture these finer details, offering more actionable insights for language learners.

Methods

The system employs a multi-stage approach:

  1. English Speech Recognition (ASR): A core component, this stage uses IBM's HMM-based ASR system. To improve accuracy for Chinese accented English speakers, they combine a database of accented English speech with native English data to build acoustic models. They also utilize constrained language modeling (LM) by combining a general LM with one built from testing sentences.
  2. Pronunciation and Prosody Assessment:
    • Pronunciation: Beyond standard ASR posterior probabilities, the system uses a special forced alignment technique. This involves generating alternative phonetic spellings based on common pronunciation errors made by Chinese speakers and then finding the best match to the actual articulation.
  3. Word Stress: Features like syllable duration, energy, and pitch are extracted. A Bayesian classifier is used to determine stressed syllables, comparing the result against a dictionary. They specifically use vowel duration and maximum loudness as key features.
  4. Rhythm: This is assessed by analyzing silence positions. A prosody phrase structure prediction model, trained on a TTS database, is used to predict pause positions. The system compares actual silences with these predicted positions to identify rhythm errors.
  5. Fluency: This is measured by combining speaking rate and the number of improper pauses.
  6. Summarization of Assessment: A Support Vector Machine (SVM) is used to fuse the various features (word error rate, posterior probabilities, word stress, rhythm, and fluency scores) into a single, comprehensive assessment score. This stage is particularly relevant for automatic language assessment agents.

Datasets

  • Chinese accented English speech database: A newly collected dataset comprising 100 subjects, 30,000 utterances, and over 70 hours of audio. This was used to improve ASR accuracy for this specific accent.
  • Native English speech data: Used in conjunction with the accented data to build acoustic models.
  • English TTS database: Manually labeled with prosody structure information, used to train the prosody phrase model for rhythm assessment.
  • 90K-word dictionary: Used for generating ground truth for word stress.
  • Test sets: Specific test sets were constructed for evaluating pronunciation, word stress, and rhythm. For rhythm evaluation, a decision tree was built using 3,232 training sentences and tested on 355 sentences.

Key Insights and Ideas

  • Beyond Phonemes: The system's strength lies in moving beyond simple phonetic accuracy to analyze more subtle but crucial aspects of spoken language like word stress and rhythm, which significantly impact intelligibility and naturalness.
  • Accent-Specific ASR: Recognizing that native-speaker ASR models are insufficient for accented speech, the paper highlights the importance of incorporating accent-specific data for acoustic modeling.
  • Prosody Phrase Prediction for Rhythm: Using TTS prosody models to predict pause positions is an innovative way to objectively assess speech rhythm, which is otherwise difficult to quantify.
  • Feature Fusion for Holistic Assessment: Combining diverse features (acoustic, prosodic, error rates) into a single score via SVM provides a more comprehensive evaluation than any single metric alone.

Key Results

  • Word Posterior Probability: A large difference in average word posterior probability was observed between utterances labeled "bad" and "fair" by human annotators, suggesting its utility in distinguishing lower proficiency levels.
  • Word Stress: Using syllable duration and maximum loudness as features achieved a classification rate of 79.71% for word stress. Adding pitch (maximum semitone) offered only a marginal improvement, suggesting duration and energy are primary correlates.
  • Rhythm: The prosody phrase prediction model achieved an accuracy rate of 83.9% and a recall rate of 75.6% in identifying rhythm errors.
  • Comparison with Human Annotation: The paper presents tables showing how automatic assessment results (e.g., for rhythm) correlate with human judgments, indicating promising levels of agreement.

Limitations and Future Work

  • Data Annotation: A significant limitation acknowledged is the lack of detailed annotations for language assessment experiments. The authors have made a first step by creating the Chinese accented English dataset.
  • Future Work: The authors plan to add richer annotations to their dataset and further develop the system. This suggests potential for incorporating more linguistic features, expanding accent coverage, and refining the assessment algorithms.

Summary updated

BibTeX
@inproceedings{shi2010spoken,
  author = {Qin Shi and Kun Li and ShiLei Zhang and Stephen M. Chu and Ji Xiao and ZhiJian Ou},
  title = {{Spoken English assessment system for non-native speakers using acoustic and prosodic features}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure