Evaluating Automatic Dialogue Strategy Adaptation for a Spoken Dialogue System

Jennifer Chu-CarrollNAACL 2000

TL;DR

This paper introduces MIMIC, a spoken dialogue system that automatically adapts its dialogue strategies based on user input and dialogue history. Experiments show that MIMIC's adaptive, mixed-initiative approach leads to more satisfied users, more efficient dialogues, and better resolution of conversational anomalies compared to less adaptive systems.

What Problem Does This Paper Address?

Modern spoken dialogue systems, while increasingly sophisticated, often rely on fixed dialogue strategies that are only locally responsive to user input. This can lead to difficulties in handling conversational anomalies, where a user might deviate from expected behavior or the system might misinterpret input. The authors identify a gap in systems that can automatically adapt their dialogue strategies, particularly in a mixed-initiative setting where both the user and system can take the lead. They aim to evaluate whether such automatic adaptation, combined with a mixed-initiative approach, improves user experience and dialogue efficiency.

Methods

The core of the paper is the evaluation of MIMIC, a mixed-initiative spoken dialogue system. MIMIC's key innovation is its "initiative module," which is decoupled from the main dialogue manager. This module uses Dempster-Shafer theory to dynamically determine the distribution of "task initiative" (leading the problem-solving) and "dialogue initiative" (leading the conversational flow) based on cues from user utterances and dialogue history.

Two control systems were created to isolate the effects of MIMIC's features:

  • MIMIC-SI (System Initiative): This system defaults to system initiative, similar to many traditional dialogue systems.
  • MIMIC-MI (Mixed Initiative, Non-Adaptive): This system is mixed-initiative but does not adapt its strategies based on dialogue history, only local context.

The evaluation involved comparing MIMIC against these controls across three dimensions: performance features (task success, efficiency, usability), discourse features (user utterance characteristics, detected cues), and initiative distribution.

Datasets

The paper describes experiments conducted with human subjects interacting with the dialogue systems. The dialogues were collected and then analyzed. The authors do not explicitly mention using pre-existing datasets for training; rather, they describe an "iterative training procedure on a corpus of transcribed and annotated human-human dialogues" to obtain the parameters for their initiative module. The experiments themselves generated a dataset of 112 dialogues with approximately 2,800 turns.

Key Insights and Ideas

  • Automatic Adaptation Matters: The most significant finding is that MIMIC's ability to automatically adapt its dialogue strategies significantly improves user satisfaction and dialogue efficiency. This suggests that users benefit from systems that can dynamically adjust their behavior.
  • Mixed Initiative + Adaptation = Better User Experience: The combination of allowing users to take initiative and the system's ability to adapt its own strategy leads to dialogues that better match user expectations and resolve problems more smoothly.
  • Handling Anomalies: MIMIC's adaptation is particularly effective at resolving dialogue anomalies, such as when users are unsure what to say or when the ASR makes errors. The system can proactively offer assistance or adjust its strategy to guide the user.
  • Initiative as a Diagnostic Tool: The analysis of initiative distribution revealed that MIMIC tended to take over task initiative when ASR performance was poor, suggesting it uses initiative shifts to compensate for system limitations and constrain user input.

Key Results

  • User Satisfaction: Users were more satisfied with MIMIC compared to both MIMIC-SI and MIMIC-MI. For MIMIC vs. MIMIC-SI, user satisfaction was 21.9 vs. 19.8 (p=0.0447). For MIMIC vs. MIMIC-MI, it was 29.5 vs. 24.4 (p=0.0364).
  • Dialogue Efficiency: MIMIC led to fewer user turns and less elapsed time. For MIMIC vs. MIMIC-SI, user turns were 10.3 vs. 13.6 (p=0.0075) and elapsed time was 229.5 vs. 277.5 (p=0.0162). For MIMIC vs. MIMIC-MI, user turns were 10.3 vs. 14.3 (p=0.0199) and elapsed time was 200.6 vs. 246.4 (p=0.0457).
  • Dialogue Quality: MIMIC showed a significantly lower ASR timeout rate compared to MIMIC-MI (5.7% vs. 15.6%, p=0.001).
  • Anomaly Resolution: MIMIC was more effective at resolving dialogue anomalies, with significantly fewer "NoNewInfo" cues and more "AmbiguousActionResolved" cues compared to MIMIC-MI.

Limitations and Future Work

  • Task Success: The study found no significant difference in task success rates across systems. The authors attribute this to the simplicity of the movie information domain, where tasks were generally easy to complete.
  • ASR Performance: While MIMIC's adaptation helped resolve issues, it did not significantly improve ASR performance itself. However, the authors suggest that the ability to adapt dialogue strategies when ASR performance is poor could enable the system to employ dialogue-strategy-specific language models for ASR in the future.
  • Future Work: The paper implies that further exploration of how initiative distribution correlates with other features, and how to leverage adaptation for ASR improvement, are potential future directions. The authors also note that their analysis of initiative distribution could be further extended.

Summary updated

BibTeX
@inproceedings{chucarroll2000evaluating,
  author = {Jennifer Chu-Carroll},
  title = {{Evaluating Automatic Dialogue Strategy Adaptation for a Spoken Dialogue System}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2000},
}
Figure