Abstract
Researchers at CMU’s Project LISTEN developed "Evelyn," an automated reading coach that uses speech recognition to follow a student’s oral reading and identify errors. By adapting the Sphinx-II recognizer to handle the disfluencies of early readers, the system achieves error detection rates significantly better than random chance. This work addresses the technical challenges of tracking disfluent speech, marking a major step toward interactive literacy tools that provide immediate, human-like feedback.
Introduction
While educational software for reading has existed for decades, most tools are "deaf"—they can display text but cannot hear if a student is struggling. Human tutors provide immediate, specific feedback when a child gets stuck or misreads a word, which is critical for developing fluency. The technical challenge lies in the nature of "disfluent" speech. Standard Automatic Speech Recognition (ASR) is designed for fluent adults. Children learning to read often hesitate, "sound out" words, repeat phrases, or skip lines entirely, breaking the linear assumptions of traditional ASR. The authors aim to bridge this gap by creating a system that can track a reader's position in a known text and accurately flag "missed words"—words the reader failed to produce correctly even after multiple attempts.
The Evelyn System
The prototype system, named Evelyn, integrates two primary components: the Sphinx-II speaker-independent continuous speech recognizer and the ORATOR speech synthesizer for generating pronunciations of out-of-vocabulary words.
To handle the complexities of oral reading, the researchers implemented several key innovations:
- Custom Language Model: Instead of a standard N-gram model, they used a probabilistic word-pair grammar that accounts for three behaviors: reading the next word correctly, skipping/repeating (transitioning to an arbitrary word in the text), and "sounding out" (transitioning to individual phonemes).
- Heuristic Text-Following: To solve "hindsight dependency"—where a recognizer changes its mind about a word once it hears the subsequent context—they implemented an "inertia" algorithm. The system waits until at least two consecutive words are recognized before moving the UI cursor.
- Pedagogical Criterion: The system focuses on "missed words"—words not read correctly at least once. This acknowledges that self-correction is a natural part of the learning process.
Experimental Evaluation
The system was evaluated using a unique Corpus of Oral Reading consisting of recordings from 27 second-graders reading 99 passages. Unlike standard ASR datasets, this corpus includes realistic classroom background noise and "non-task" speech.
The task was framed as alignment rather than transcription. Since the text on screen is known, the goal is to align the audio signal to the text while allowing for "illegal" deviations like skips and errors. The system’s performance was compared against human-transcribed "gold standard" logs.
As shown in the results, while raw ASR accuracy was low (28.1% error rate) due to the difficulty of children's speech, the system excelled at missed-word detection. Evelyn achieved a coverage of 63.6% and a precision of 60.9%. This performance is roughly 25 times better than a random baseline algorithm based on the average student disfluency rate of 2.5%.
Discussion and Future Work
Despite its success, Evelyn faced several limitations. At the time, the system required significant processing power (100+ MIPS) to run near real-time. Furthermore, the use of adult female phonetic models likely limited accuracy; training specifically on children's voices is a clear path for improvement. Future work also involves refining the intervention logic—determining exactly when and how the "coach" should provide feedback—and reducing the UI lag caused by hindsight dependency.
Summary updated