A framework for dialogue data collection with a simulated ASR channel

Matthew N. Stuttle, Jason D. Williams, Steve YoungInterspeech 2004

TL;DR

This paper introduces a novel "Wizard of Oz" framework that simulates Automatic Speech Recognition (ASR) errors to collect realistic dialogue data for training machine learning-based dialogue managers. This approach addresses the limitations of existing data collection methods by allowing for controlled ASR error rates, crucial for building robust dialogue systems.

What Problem Does This Paper Address?

Developing effective dialogue management for spoken dialogue systems (SDSs) is challenging. Traditional rule-based systems are time-consuming to author and may not be optimal. Machine learning (ML) approaches offer a promising alternative, but they require large amounts of training data. Collecting this data presents a dilemma: human-human conversations don't reflect the ASR channel, while human-computer dialogues are constrained by the existing system's policy. This paper tackles the problem of acquiring dialogue data that accurately mimics the experience of interacting with an SDS, including the inevitable errors introduced by ASR.

Methods

The core of the proposed framework is a modified "Wizard of Oz" setup. Two participants, a "subject" and a "wizard," communicate in separate rooms. The subject hears the wizard directly, but the wizard only sees the subject's utterances as simulated ASR output on a screen. This simulation is achieved by having a typist transcribe the subject's speech, which is then fed into an ASR confusion system. This system introduces errors based on a weighted finite state transducer (WFST) model, allowing for adjustable Word Error Rates (WERs). The framework also incorporates a turn-taking model and internal states to manage the dialogue flow.

Datasets

  • ATIS air travel information corpus: Used for evaluating the ASR confusion system's performance.
  • TIMIT corpus: Used to train the phonetic confusion matrix for the ASR simulation.

The paper does not introduce a new benchmark dataset but rather a framework for collecting data that can be used to train ML models.

Key Insights and Ideas

  • Simulating ASR errors is crucial: Directly collecting human-computer dialogue data is limited by the system's fixed policy. Human-human data lacks the critical ASR channel imperfections. Simulating ASR errors allows for controlled experimentation and data collection that better reflects real-world SDS interactions.
  • WFSTs offer a principled way to simulate ASR errors: By modeling the speech recognition process with WFSTs, the authors can generate realistic confusions that mimic those produced by actual ASR systems, going beyond simple random error injection.
  • Adjustable WER is key for robustness: The ability to vary the ASR error rate (WER) in the simulation is vital for training dialogue managers that can handle a range of recognition performance, which is often uncertain in real systems.
  • Wizard comfort improves with practice: The authors observed that wizards become more comfortable and efficient with the interface after conducting a few dialogues, highlighting the importance of sufficient data collection sessions.

Key Results

The evaluation showed that the WFST-based simulated ASR data closely matched the performance degradation observed with real ASR data on the ATIS corpus. Specifically, the F-measure (a metric for concept slot retrieval) degraded linearly with increasing WER for both real and simulated data. The simulated data tended to yield slightly higher F-measures for comparable WERs. A "naive" corruption system performed significantly worse than both real and WFST-simulated data, indicating the importance of a principled error generation model. The paper reports that the end-to-end perceived recognition times in their setup are not unrealistic for an ASR system.

Limitations and Future Work

Limitations: * The current framework relies on a human typist, which introduces a delay compared to a real-time ASR system. * The wizard's ability to process information is limited compared to a computer algorithm (e.g., confidence scores, parse scores). * The naive corruption system's performance was significantly worse, suggesting that simply injecting random errors is not sufficient.

Future Work: * Expanding the wizard's interface to support "point and click" interactions. * Exploring alternative forms of presentation to the wizard, such as parse results and confidence scores. * Investigating the use of this framework for multi-modal interfaces. * Further analysis of collected data for user modeling and direct analysis.

Summary updated

BibTeX
@inproceedings{stuttle2004framework,
  author = {Matthew N. Stuttle and Jason D. Williams and Steve Young},
  title = {{A framework for dialogue data collection with a simulated ASR channel}},
  booktitle = {Interspeech},
  year = {2004},
}
Figure