TL;DR
This paper investigates how different dialog control strategies (system-led, mixed, user-led) and interface types (telephony vs. multi-modal kiosk) affect performance and user satisfaction in a large directory information access task. Surprisingly, dialog initiative had little impact on accuracy or task completion, but the multi-modal kiosk interface significantly improved user experience and reduced dialog length.
What Problem Does This Paper Address?
Spoken dialog systems are becoming increasingly sophisticated, but understanding how to best design their interaction flow remains a challenge. Specifically, the role of "dialog initiative" – who controls the conversation, the system or the user – is crucial. Previous research suggested system-led dialogs were generally better for performance and acceptance. However, these studies often focused on voice-only interfaces. This paper addresses the gap of understanding how dialog initiative, combined with the richer capabilities of multi-modal interfaces (like kiosks with screens and touch input), impacts user interaction for a complex task like accessing a large directory.
Methods
The researchers designed an experiment to compare three dialog initiative strategies:
- System Initiative (SI): The system strictly guides the user, and the speech recognition grammar is tightly constrained to expected responses.
- Mixed Initiative (MI): The system guides, but the speech recognition grammar is more flexible, allowing users to provide extra information.
- User Initiative (UI): The system prompts are open-ended, and the grammar is flexible, similar to MI.
These strategies were tested on two interfaces:
- Telephony-only: Voice input and output only.
- Multi-modal Kiosk: Speech input/output combined with a graphical display and touch input.
The experiment involved 50 AT&T Labs employees performing six tasks each, including information queries and placing a call. They collected objective data (concept accuracy, in-grammar utterances, dialog turns) and subjective data (perceived task completion, ease of use, user satisfaction).
Datasets
The experiment used a subset of the AT&T corporate directory containing 4361 employees. The lexicon for the mVPQ application was expanded to 54,626 entries to account for variations like nicknames and last names. No new benchmark datasets were introduced; existing internal data was utilized.
Key Insights and Ideas
- Dialog Initiative Matters Less Than Expected: Contrary to some prior work, the study found that dialog initiative strategy had surprisingly little impact on objective measures like concept accuracy or subjective measures like perceived task completion and user satisfaction for this specific task.
- Multi-modal Wins for User Experience: The kiosk interface, with its ability to display information visually and accept touch input, significantly outperformed the telephony-only interface in terms of user satisfaction and ease of use.
- Efficiency Gains with Multi-modal: Dialogs were substantially shorter on the kiosk interface, largely due to users effectively using the touchscreen for disambiguation, demonstrating a fluid switch between speech and touch input.
- Grammar Flexibility Trade-offs: While User Initiative (UI) led to fewer in-grammar utterances, it didn't negatively impact overall concept accuracy, suggesting that larger, more flexible grammars don't necessarily hinder performance in this context.
Key Results
- Concept Accuracy: No significant differences in overall concept accuracy were found across dialog initiative strategies.
- User Satisfaction & Ease of Use: The kiosk interface received significantly higher ratings for user satisfaction (1.5 vs. 2.7 on a 7-point scale) and ease of use (1.9 vs. 3.3) compared to the telephony interface.
- Dialog Length: Kiosk dialogs averaged 3.6 turns per task, significantly shorter than telephony dialogs at 5.4 turns.
- In-grammar Utterances: The User Initiative (UI) strategy resulted in a significantly lower proportion of in-grammar utterances compared to System Initiative (SI) and Mixed Initiative (MI).
- Task Completion: Perceived task completion was higher for the kiosk interface across most tasks.
Limitations and Future Work
The authors acknowledge that the complexity of the task domain (directory access) might be relatively low, which could explain why dialog initiative had less impact than in studies of more complex tasks. They also note that the interpretation of "task completion" could be ambiguous, particularly in cases where information was not available. Future work could explore these effects in more complex domains and investigate how users adapt to different initiative strategies in multi-modal environments. The study also suggests that the ability to seamlessly switch between speech and touch input in multi-modal systems is a key factor in improving user experience and efficiency.
Summary updated