Intelligently Aiding Human-Guided Correction of Speech Recognition

Keith Vertanen, Per Ola KristenssonAAAI 2010 · DOI

Introduction

Speech recognition on mobile devices remains a significant challenge due to acoustic noise and limited computational resources. A primary pain point is the "correction phase"; when a system misrecognizes a word, attempting to fix it via voice often leads to cascading errors, where the correction itself is misidentified. This research introduces Parakeet, a mobile interface designed to shift the focus from perfect initial recognition to rapid, human-guided error correction using multi-modal touch inputs.

System Design and Principles

Parakeet was developed for the Nokia N800 mobile device, guided by three principles: avoiding cascading errors through touch-based fallbacks, exploiting the recognizer's internal hypothesis space, and supporting fragmented, mobile interaction.

At the heart of the interface is the Word Confusion Network (WCN). Rather than displaying a single "best guess" sentence, Parakeet presents a grid where the top row represents the most likely transcription, and the columns beneath it offer alternative hypotheses generated by the speech engine.

The Parakeet interface on a Nokia N800, showing the word confusion network grid and touch interaction.

Multi-modal Correction Features

To facilitate fast editing, the system supports several touch gestures:

  • Tapping: Users can simply select an alternative word from the WCN column to replace an error.
  • Crossing: A swiping gesture allows users to delete or modify multiple words in a single motion.
  • Copying: Words can be dragged and dropped to different positions in the sentence.
  • Morphological Variants: To improve intuitiveness, the system offers related word forms (e.g., "constitute" vs. "constitutional") using the Snowball stemmer, providing a more human-centric alternative to purely acoustic predictions.
  • Predictive Keyboard: For errors where the correct word is missing from the WCN, a software keyboard provides completions to minimize typing effort.

Experimental Evaluation

The researchers conducted "oracle" simulations to determine the optimal interface density. These simulations demonstrated that the majority of potential error reduction is achieved by showing only the first few alternatives, allowing the UI to maintain large, touch-friendly buttons.

Oracle simulations showing the impact of confusion network cluster size and specific features on Word Error Rate (WER).

Following simulations, user studies were conducted across three scenarios: indoor newswire entry, outdoor (walking) newswire entry, and mobile voice search queries.

Key Results

  • Error Reduction: In newswire tasks, initial Word Error Rates (WER) of 16% (indoors) and 25% (outdoors) were reduced to just 1% and 2% respectively after user correction.
  • Entry Speeds: Users achieved entry speeds of 18 wpm indoors and 13 wpm outdoors. While slower than desktop typing, these rates are competitive with traditional mobile input methods like T9 while requiring less manual effort.
  • Interface Efficiency: Users were able to resolve 96% of errors using only the confusion network, rarely needing to resort to the manual predictive keyboard. In voice search tasks, users reached a rate of 1.7 characters per second, significantly outperforming contemporary mobile search benchmarks.

Discussion and Future Work

The study identified system latency as the primary bottleneck; a mean recognition delay of 22 seconds significantly hindered the user experience. Despite this, the results suggest that leveraging the recognizer's hypothesis space through a touch-based WCN is a highly effective strategy for mobile text entry. Future work will focus on reducing processing time and refining the interface to remain efficient across varying levels of initial recognition accuracy.

Summary updated

BibTeX
@inproceedings{vertanen2010intelligently,
  author = {Keith Vertanen and Per Ola Kristensson},
  title = {{Intelligently Aiding Human-Guided Correction of Speech Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2010},
  doi = {10.1609/aaai.v24i1.7712},
}
Figure