A comparison of confirmation styles for error handling in a speech dialog system

Hirohiko Sagawa, Teruko Mitamura, Eric NybergInterspeech 2004 · OpenReview

Abstract

Speech recognition errors remain a significant challenge in voice user interfaces, often leading to dialog breakdowns. This paper evaluates three confirmation strategies—explicit, final, and implicit—to determine how they impact user satisfaction and efficiency. Through a controlled experiment using the CAMMIA dialog system, the study reveals a contextual shift in user preference: while efficient "final" confirmations are preferred when recognition is accurate, users favor the reliability of "explicit" confirmations when errors occur.

Introduction

In the early development of speech dialog systems, word error rates were a primary barrier to usability. When a system misinterprets a user, it creates a "dialog breakdown" that requires correction. To address this, systems employ various confirmation styles to verify information. While prior research focused heavily on efficiency—minimizing the number of turns to complete a task—this study investigates the gap between speed and user satisfaction, particularly in erroneous situations.

Example of a dialog breakdown where a system error leads to a correction utterance.

Methods and System Design

The researchers utilized the CAMMIA system, a VoiceXML-based dialog manager, to implement three distinct confirmation styles for a weather inquiry task:

  1. Explicit Confirmation: The system verifies each piece of information immediately (e.g., "Did you say Tokyo?").
  2. Final Confirmation: The system collects all required slots (e.g., area and date) before confirming the entire set at the end.
  3. Implicit Confirmation: The system repeats the recognized information while simultaneously prompting for the next slot (e.g., "Tokyo. And for what date?").

To facilitate error recovery, the system uses dynamically generated correction grammars. When a potential error is detected, the system generates a grammar on the fly to recognize correction utterances (such as "No, I said Tokyo") based on the conversational history.

Experimental Setup

The study involved 13 Japanese subjects performing a weather information task with 20 different combinations of area and date. To ensure sufficient data on error handling, the researchers intentionally configured the speech recognizer to an accuracy of approximately 70%. This allowed for a robust comparison of how each confirmation style performed when the system failed to recognize user input correctly.

Key Results and Discussion

The experimental results highlight a clear trade-off between efficiency and perceived safety. As shown in the performance metrics, implicit confirmation was the most efficient in terms of time and turn count, while explicit confirmation was the slowest.

Efficiency metrics comparing the number of turns and dialog time across the three confirmation styles.

However, efficiency did not correlate directly with user preference:

  • The "Happy Path" (No Errors): When the system recognized input correctly, users strongly preferred Final Confirmation (Average Rank: 1.64). It felt less repetitive and more natural for straightforward tasks.
  • The "Error Path" (With Errors): When errors occurred, Explicit Confirmation became the clear winner (Average Rank: 1.59). Users found it easier to isolate and fix individual errors one at a time. Conversely, Implicit Confirmation was ranked lowest in these scenarios (Average Rank: 2.41), as users found it difficult to interrupt the system's flow to correct information that had already been integrated into a subsequent prompt.

Conclusion

There is no single "best" confirmation style for all scenarios. User preference is highly dependent on the system's perceived reliability at any given moment. The findings suggest that future speech interfaces should be adaptive: utilizing efficient final confirmations by default but automatically switching to explicit, "hand-holding" styles the moment a correction utterance or low-confidence recognition is detected.

Summary updated

BibTeX
@inproceedings{sagawa2004comparison,
  author = {Hirohiko Sagawa and Teruko Mitamura and Eric Nyberg},
  title = {{A comparison of confirmation styles for error handling in a speech dialog system}},
  booktitle = {Interspeech},
  year = {2004},
  url = {https://openreview.net/forum?id=_7vok9i5T5e},
}
Figure