TL;DR
This paper introduces "distilling dialogues," a novel method for transforming natural human-human conversations into realistic human-computer dialogues. This technique aims to bridge the gap between naturally occurring language and the language required for effective dialogue systems, offering a valuable supplement to existing methods like Wizard-of-Oz studies.
What Problem Does This Paper Address?
Developing natural language dialogue systems requires realistic conversational data. However, natural human-human dialogues often differ significantly from how users would interact with a computer. Users might have different expectations, use more informal language, or engage in conversational tangents not relevant to a system's function. Conversely, Wizard-of-Oz (WoZ) studies, where a human simulates a system, can be resource-intensive and may not fully capture genuine user needs and behaviors due to the artificiality of the setup. This paper addresses the challenge of obtaining dialogue corpora that accurately reflect human-computer interaction by proposing a method to "distill" existing human dialogues into a more computer-appropriate format.
Methods
The core method proposed is "distilling dialogues." This involves post-processing naturally occurring human-human dialogues to make them resemble human-computer interactions. The process requires an understanding of the target dialogue system's capabilities and limitations. Key guidelines for distillation include:
- System Utterances: These are modified to provide all relevant information at once, mimicking a system's ability to access and present information efficiently. Irrelevant or conversational filler is removed.
- User Utterances: These are modified minimally, primarily as a consequence of system utterance changes. Utterances that are no longer valid due to system changes are removed.
- Non-Dialogue Contributions: Utterances from other speakers, noise, or interruptions are removed to maintain focus on the core interaction.
- Computer-like Behavior: The system is made to behave like a computer, meaning it doesn't engage in human-like behaviors such as jokes, irony, or unnecessary repetition.
The authors contrast this with traditional methods: analyzing real human dialogues (good for user goals but not language) and WoZ studies (good for interaction but artificial context). Distilling dialogues aims to combine the ecological validity of real dialogues with the computer-like interaction of WoZ, but through a more controlled, post-hoc process.
Datasets
The paper uses a corpus of 39 distilled dialogues from telephone conversations between human information providers and callers seeking local bus timetable information. These dialogues were collected at Linköping University. The authors note that the original dialogues were in Swedish.
Key Insights and Ideas
- Bridging the Gap: Distilling dialogues offers a practical way to create more realistic training data for dialogue systems by transforming existing, ecologically valid human conversations.
- Efficiency: The distillation process is relatively fast (about three hours for 39 dialogues) and results in significantly shorter dialogues (average 27% reduction), making them more manageable for system development.
- System vs. Human Language: The core difference lies in how information is presented and how conversational "noise" is handled. Distillation emphasizes directness and relevance, mirroring system design principles.
- Limitations of WoZ: The paper highlights potential distortions in WoZ studies due to human operators trying to act "computer-like," which can be difficult and introduce systematic errors. Distillation, being a post-hoc process, can be more consistent.
- User Behavior Nuances: The study found that the amount of distillation (and thus the removal of "unnatural" dialogue elements) didn't strongly correlate with the user's task, suggesting that user interaction style is influenced by factors beyond just the task itself.
Key Results
The distillation process resulted in dialogues that were, on average, 27% shorter. In some cases, up to 73% of the original dialogue was removed, particularly in dialogues containing discussions outside the application's scope (e.g., finding parking). The authors added a small number of utterances (five system, five user) to ensure dialogue continuity where necessary, noting that using audio tapes instead of transcripts might have reduced this need. They also acknowledge that the distillation was primarily for speech-only systems, as accounting for multi-modal interaction would add further complexity.
Limitations and Future Work
The authors acknowledge several limitations:
- Artificiality: While aiming for realism, the distillation process itself introduces a form of artificiality by imposing a specific model of computer interaction. The resulting dialogues are less accurate as a model of the entire dialogue flow compared to a final system, but are good approximations for smaller segments.
- Multi-modality: The current distillation focused on speech-only systems. Future work could involve a second distillation pass to incorporate multi-modal aspects.
- System Model Dependency: The distillation process relies on an assumed model of the dialogue system's capabilities. Different assumptions would lead to different distilled dialogues.
Future work could involve further distillation to focus on multi-modal interaction and exploring how different system models influence the distillation process.
Summary updated