Adapting a duration synthesis model to rate children's oral reading prosody

Minh Duong, Jack MostowInterspeech 2010

TL;DR

This paper introduces a novel method to automatically assess children's oral reading prosody by adapting a speech synthesis model. This approach outperforms previous methods that relied on adult narrations, offering a more scalable and generalized way to evaluate reading fluency and comprehension.

What Problem Does This Paper Address?

Assessing children's oral reading fluency is crucial for education, but current methods are often limited. While reading rate (words per minute) is easy to measure, it ignores expressiveness. Existing methods for evaluating expressiveness, like correlating children's prosody with adult narrations, are labor-intensive and require a specific adult recording for every sentence. This paper aims to bridge this gap by developing an automated system that can assess oral reading prosody without needing an adult narration of each sentence, making it more practical for large-scale educational applications.

Methods

The core of the paper is adapting a duration synthesis model for assessment.

  • Standard Approach (Synthesis): Speech synthesis models typically learn to predict the duration of phones (basic units of sound) based on linguistic context. They use decision trees trained on large corpora of adult speech, where each leaf node in the tree represents a specific phonetic context and stores the mean duration for that context.
  • Novel Approach (Assessment): The authors adapt this synthesis model by leveraging both the mean and the standard deviation of durations stored at each leaf node. Instead of using the mean to prescribe a duration, they use it to calculate the likelihood of a child's actual phone duration. This allows them to quantify how well a child's reading matches the "normative" prosody learned from multiple adult speakers.
  • Feature Engineering: The duration model uses a rich set of features, including phone identity, position within the syllable and word, syllable structure, and word-level features like part-of-speech and number of syllables.
  • Aggregation: Seven different formulas are proposed to aggregate phone-level prosody ratings into an overall utterance-level rating. These include average log-likelihood, z-scores, root mean squared error, mean absolute error, and various correlation measures at both phone and word levels.

Datasets

  • Training Data: A corpus of adult-narrated speech used in the Project LISTEN Reading Tutor. This corpus consists of 24,816 sentences and over 811,000 phones, read by 20 different adult narrators.
  • Children's Data: Oral reading data from 235 students (ages 7-10) using the Project LISTEN Reading Tutor. This data comprises over 399,000 utterances and more than 8.3 million phones.
  • Evaluation Data: Standardized tests of fluency and reading comprehension administered to the students at the beginning (pretest) and end (posttest) of a semester.

Key Insights and Ideas

  • Normative Model Advantage: Adapting a synthesis model trained on multiple adult voices creates a "normative" model that generalizes better than relying on a single adult's narration for comparison. This is because it smooths out individual speaker idiosyncrasies.
  • Value of Standard Deviation: The inclusion of standard deviation in the assessment formulas (specifically average log-likelihood and z-score) proved to be a significant predictor of student performance, highlighting that variability in prosody is as important as the average.
  • Scalability: The primary advantage is eliminating the need for adult narrations for every sentence, making automated prosody assessment far more scalable for educational tools.

Key Results

  • Predictive Power: The normative model (using the adapted synthesis model) slightly surpassed the previous correlational method (using adult narrations) in predicting students' posttest fluency and comprehension scores. Adjusted R² values were 0.572 vs. 0.369 for fluency and 0.565 vs. 0.362 for comprehension, respectively.
  • Incremental Improvement: Adding the normative ratings to pretest scores accounted for an additional ~0.01 in variance for both fluency and comprehension, demonstrating its value beyond baseline performance.
  • Feature Importance: The average log-likelihood (which uses both mean and standard deviation) was consistently selected as the most important feature by stepwise regression, underscoring its effectiveness.

Limitations and Future Work

  • Data Sparseness: The estimates of mean and standard deviation at leaf nodes might suffer from data sparseness, especially for less frequent phonetic contexts.
  • Future Work: The authors suggest implementing "deleted interpolation" to smooth sparse estimates. They also plan to extend this approach to model and assess pitch and intensity, providing a more comprehensive evaluation of oral reading prosody.

Summary updated

BibTeX
@inproceedings{duong2010adapting,
  author = {Minh Duong and Jack Mostow},
  title = {{Adapting a duration synthesis model to rate children's oral reading prosody}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure