A Robust System for Natural Spoken Dialogue

James F. Allen, Bradford W. Miller, Eric K. Ringger, Teresa SikorskiACL 1996 · DOI

Abstract

The TRAINS-95 system demonstrates that robust, natural spoken dialogue is achievable even when underlying speech recognition is unreliable. By combining statistical error correction (SPEECHPP), "minimal covering" parsing, and a "strong commitment" discourse strategy, the researchers developed a system where users solved complex route-planning tasks 30% faster than through keyboard input—despite a significant 30% word error rate.

Introduction

Building a computer capable of spontaneous, unconstrained conversation in real-time presents a formidable challenge. In the mid-1990s, most speech systems were "brittle," where a single misheard word could lead to a total dialogue collapse. Human speech is naturally filled with disfluencies like "umms," "ahhs," false starts, and grammatical fragments. The TRAINS-95 project addressed the gap between high-accuracy recognition of read text and the "wild west" of human-to-human interaction, creating a system that robustly interprets intent even when transcriptions are riddled with errors.

System Architecture

The TRAINS-95 architecture is a multi-layered pipeline designed for resilience. It moves beyond simple transcription to deep interpretation through several key components:

  1. Statistical Error Post-Correction (SPEECHPP): Before parsing, a post-processor acts as a "noisy channel" decoder. Using a word-bigram language model, it "cleans up" output from the Sphinx-II speech recognizer, guessing the user's actual intent based on common patterns in the TRAINS domain.
  2. Robust "Minimal Covering" Parsing: Instead of requiring perfect sentence structures, the bottom-up parser identifies a sequence of "speech acts" (e.g., SUGGEST, CONFIRM). If a full parse fails, it finds the shortest sequence of fragments that "covers" the input, effectively ignoring uninterpretable noise.
  3. The "Strong Commitment" Strategy: The discourse manager follows a bold philosophy: when faced with ambiguity, it makes a specific guess rather than asking for clarification. It is often easier for a user to correct a slightly incorrect plan than to repeatedly answer "What did you mean?"
  4. Integrated Reasoning: A domain-specific problem solver and "verbal reasoner" ensure that even fragmented speech acts result in a plan that respects the constraints of the map and available resources.
The TRAINS-95 system architecture, showing the flow from speech recognition to discourse management and problem solving.

Evaluation and Results

The system was evaluated using the TRAINS-95 corpus and "stress-tested" using the ATIS (Air Travel Information Service) language model, which intentionally pushed the word error rate (WER) to 30%.

Performance Metrics

Traditional metrics like WER often fail to capture the utility of a dialogue system. The addition of SPEECHPP significantly improves word accuracy across various training set sizes compared to using the recognizer alone.

Word accuracy improvements using SPEECHPP post-processing compared to baseline and augmented Sphinx-II models across different training set sizes.

Key findings include:

  • Efficiency Gain: Users solved problems using speech in only 68% of the time required for keyboard input.
  • User Preference: 75% of subjects (12 out of 16) preferred speech over keyboard for unconstrained tasks.
  • Robustness: The system successfully guided users to solutions despite the 30% WER.
  • Solution Quality: There was no significant difference in the quality of routes planned between speech and keyboard modalities.

Discussion and Key Insights

A primary insight of this work is that domain knowledge acts as a powerful filter. By understanding the state of the "world" (e.g., train locations), the system can correct speech errors that are contextually impossible. Furthermore, the study proves that speech is faster than typing even when the recognition is "broken," provided the system can handle the messiness.

Limitations and Future Work

The system's reliance on domain-specific algorithms limits its portability to other tasks without significant rewriting. Additionally, the "strong commitment" strategy occasionally led to "overshooting" destinations when rejections were misinterpreted as acceptances. Future iterations, such as TRAINS-96, aim to develop a domain-independent plan reasoning "shell" to separate conversational logic from specific tasks.

Summary updated

BibTeX
@inproceedings{allen1996robust,
  author = {James F. Allen and Bradford W. Miller and Eric K. Ringger and Teresa Sikorski},
  title = {{A Robust System for Natural Spoken Dialogue}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {1996},
  doi = {10.3115/981863.981872},
}
Figure