TL;DR
This paper introduces a novel architecture for building multi-domain spoken dialog systems (SDS) by seamlessly integrating existing, independent SDS as "black boxes." This approach significantly reduces development effort and cost by allowing the reuse of pre-trained models, enabling richer user interactions without modifying the underlying systems.
What Problem Does This Paper Address?
Developing sophisticated, multi-domain spoken dialog systems is a significant challenge. Traditional approaches often require building monolithic systems from scratch or extensively modifying existing ones to handle multiple tasks. This is time-consuming and expensive, especially given the complexity of dialog management, which can lead to an exponential increase in states when dealing with multiple domains. The paper addresses the need for a more efficient way to combine existing, specialized SDS into a unified system that can handle a broader range of user requests, making advanced conversational AI more accessible.
Methods
The core of the proposed architecture is a Task Manager that acts as a central orchestrator. It receives user input, classifies the user's intent into one of the available sub-domains, and routes the request to the appropriate single-task SDS. Crucially, the sub-systems are treated as "black boxes" and are unaware they are part of a larger multi-domain system.
To facilitate this, a Context Register plays a vital role. It stores shared information (like dates, times, and locations) across different sub-dialogs. When a sub-system is activated after a period of inactivity, the Context Register injects relevant stored information into the user's utterance, creating a "fake user turn." This allows the sub-system to receive context as if it had been actively participating in the dialog.
The Task Manager uses a Logistic Regression (LR) model for sub-task classification, trained on features like bag-of-words and bigrams. The system also incorporates a Multi-modal Answer Generator and a Synthesizer (TTS) for generating system responses.
Datasets
The paper does not explicitly mention specific, publicly available datasets used for training the individual sub-systems. However, it states that the Task Manager's LR model was trained using a total of 2000 sample sentences extracted from dialogs and the training samples of the semantic parsers of each task. The evaluation was conducted with 28 naive real users performing scenarios across one, two, or all three integrated domains.
Key Insights and Ideas
- "Black Box" Integration: Treating existing SDS as independent modules that don't need to be aware of the multi-domain context is a key enabler for efficient integration.
- Context Injection: The Context Register's ability to inject relevant information into user turns is crucial for maintaining dialog coherence and allowing sub-systems to function without modification. This effectively bridges the gap created by inactive sub-systems.
- Reduced Development Effort: The architecture significantly lowers the barrier to entry for creating multi-domain SDS, as it leverages existing work and minimizes the need for retraining or redesigning individual components.
- Scalability: The approach suggests that adding new domains is relatively straightforward, primarily involving updates to the Context Register and the Task Manager's training data.
Key Results
The proposed architecture was applied to integrate three heterogeneous SDS: a sport facility booking system, a weather service, and a personal calendar. An evaluation with 28 naive users performing 207 dialogs showed promising results:
- Task Identification Accuracy: The accuracy of the task classifier was 91.7%, comparable to the training accuracy.
- Dialog Success Rates:
- Correct dialogs with 1 task: 94.3%
- Correct dialogs with 2 tasks: 85.1%
- Correct dialogs with 3 tasks: 76.0%
- Overall correct dialogs: 84.5%
The success rate decreased as the number of tasks in a dialog increased, which is expected due to the inherent complexity of multi-task interactions.
Limitations and Future Work
The authors acknowledge that the success ratio decreases for scenarios involving more than one task. They also note that the accuracy of the task identification was primarily affected by user turns that were valid for more than one task.
Future work suggested includes:
- More Objective Evaluation: Conducting evaluations that include objective measures beyond user satisfaction.
- User Differentiation: Developing sub-systems tailored to different user types, such as a system-oriented one for naive users and a mixed-initiative one for experts.
- Improving Context Injection: Further research into the optimal format for injected context to maximize performance and avoid misunderstandings.
Summary updated