Robust tracking for automatic reading tutors

Emre Yilmaz, Dirk van Compernolle, Hugo Van hammeInterspeech 2012

TL;DR

This paper introduces a novel "phone matching" algorithm for automatically tracking a reader's position in digital reading tutors. By comparing the recognized speech to the expected phonetic transcription of words, this method significantly improves tracking accuracy and reduces distracting cursor jumps compared to previous state-based approaches.

What Problem Does This Paper Address?

Automatic Reading Tutors (ARTs) are valuable tools for helping children improve their reading skills, offering personalized feedback and practice. A crucial component of these tutors is the ability to accurately track where the child is currently reading on the screen. This tracking is essential for highlighting the target word, advancing the page, and providing timely assistance. However, existing tracking methods often struggle with the inherent variability of children's speech, leading to mistimed cursor movements. If the cursor moves too quickly, it can be jumpy and distracting; if it moves too slowly, it can hinder the reading flow and assessment accuracy. This paper aims to address this challenge by developing a more robust and accurate tracking algorithm.

Methods

The authors propose a new tracking algorithm that moves beyond simply relying on the "recognizer's estimate" (the most likely HMM state). Instead, their core innovation is a phone matching (PM) criterion. This method calculates the phonetic distance between the recognized speech and the phonetic transcription of the target word using a dynamic programming (DP) algorithm. This allows for a more nuanced evaluation of how well a reading attempt matches the expected pronunciation, even if it's not a perfect match.

They also introduce an Advanced PM algorithm that combines the phone matching approach with the recognizer's estimate. This hybrid approach acts as a fallback mechanism: if the phone matching criterion fails to detect a sufficiently close match (e.g., due to significant reading errors or miscues), the system uses the recognizer's estimate to prevent the tracker from getting stuck. Specifically, if the recognizer consistently identifies a word beyond the current reading position for a short duration, the tracker will advance to compensate for potential tracking loss.

Datasets

The paper utilizes the CHOREC database, which contains reading sessions from 400 Dutch-speaking elementary school children. This database includes 279 reading sessions with 9 graded text stories, manually annotated with word boundaries and information about reading miscues. This dataset was crucial for estimating ideal cursor movement instants and evaluating the performance of the proposed trackers.

Key Insights and Ideas

  • Phonetic similarity is key: Directly comparing recognized phones to the phonetic transcription of target words provides a more robust measure of reading progress than relying solely on HMM states.
  • Dynamic programming for flexible matching: The DP algorithm allows for flexible alignment of recognized speech to target transcriptions, accommodating insertions, deletions, and substitutions, which are common in children's reading.
  • Hybrid approach for robustness: Combining phone matching with the recognizer's estimate creates a more resilient system that can handle both minor speech variations and significant reading errors without losing track.
  • Defining new evaluation metrics: The authors introduce "Timing Accuracy of Tracking" (TAT) and "Session Timing Accuracy" (STAT) to provide a more comprehensive evaluation of tracking performance, going beyond simple error rates to capture the nuances of timing and jumpiness.

Key Results

The proposed Advanced PM algorithm demonstrated significantly superior performance compared to the state-based tracker.

  • Timing Accuracy of Tracking (TAT): The Advanced PM achieved a TAT of 81.03%, a substantial improvement over the state-based approach's 32.36%.
  • Session Timing Accuracy (STAT>70%): The Advanced PM tracker achieved a STAT>70% in 244 out of 279 sessions, indicating that in most sessions, the tracker was well-timed for over 70% of the words. In contrast, the state-based algorithm only achieved this in 3 sessions.
  • Jumpiness: The Advanced PM tracker also showed better performance in terms of "jumpiness" (measured by VL and VE values), indicating fewer instances of the tracker being stuck or excessively jumpy.

Limitations and Future Work

  • Limitations: The paper notes that the simplified PM algorithm can get stuck easily due to poor reading or skipping words, highlighting the necessity of the advanced hybrid approach. The evaluation metrics, while improved, still rely on manual annotation of the CHOREC database.
  • Future Work: The authors suggest that the advanced PM tracker is perceived to be better not only for correct readings but also for readings with many skips, recaptures, and hesitations, implying potential for further exploration in these complex scenarios. The paper also mentions that the threshold for accepting a reading attempt is set based on the number of phones in the target transcription, suggesting that further tuning of this threshold could be beneficial for different types of reading tasks or age groups.

Summary updated

BibTeX
@inproceedings{yilmaz2012robust,
  author = {Emre Yilmaz and Dirk van Compernolle and Hugo Van hamme},
  title = {{Robust tracking for automatic reading tutors}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure