A Hierarchical Domain Model-Based Multi-Domain Selection Framework for Multi-Domain Dialog Systems

Seonghan Ryu, Donghyeon Lee, Injae Lee, Sangdo Han, Gary Geunbae Lee, Myungjae Kim, Kyungduk KimCOLING 2012

TL;DR

This paper introduces a novel Hierarchical Domain Model-based Multi-Domain Selection Framework (HDM-MDSF) that allows dialog systems to simultaneously handle multiple domains, a significant improvement over traditional systems that are limited to a single domain at a time. The HDM-MDSF achieves this by statistically identifying candidate domains and then heuristically selecting final domains, leading to more accurate and faster dialog services.

What Problem Does This Paper Address?

Traditional multi-domain dialog systems are designed to serve only one domain at a time. This limitation becomes problematic when users interact with systems that naturally span multiple related domains, such as a TV program and video-on-demand (VOD) service. For instance, a user asking about "animation programs" might be interested in both TV shows and VOD movies. Current systems struggle to identify and manage these overlapping interests, often requiring users to explicitly select a domain or leading to less efficient interactions. This paper addresses the challenge of enabling dialog systems to accurately identify and serve one or more relevant domains concurrently, enhancing the user experience in complex, multi-domain scenarios.

Methods

The core of the proposed solution is the Hierarchical Domain Model (HDM), which formally describes domain capabilities and their hierarchical relationships. This HDM is integrated into a Multi-Domain Selection Framework (MDSF). The MDSF operates in two main stages:

  1. Candidate Domain Detection: This component statistically identifies one or more potential domains that a user's utterance might relate to. It leverages the HDM to understand domain capabilities and relationships.
  2. Final Domain Determination: This component heuristically selects one or more final domains from the candidates. It considers the dialog history and the type of dialog act in the user's utterance, again guided by the HDM.

The HDM itself defines dialog acts with specific characteristics: MULTIPLE (can be served by multiple domains), SINGLE (served by only one domain), and ARBITRARY (served by one domain, with results being equivalent across domains). The framework also introduces an automatic multi-domain label annotation process for training data, which is crucial for handling utterances that can belong to multiple, closely related domains.

Datasets

The paper mentions using corpora for three base domains: TV program, VOD, and TV device. For evaluating the candidate domain detection component, they used a dataset consisting of 2628 user utterances, with 52.6% belonging to a single domain and the rest to multiple domains. These multi-domain labels were hand-annotated for evaluation. The paper does not explicitly name or introduce new benchmark datasets but rather utilizes existing data for the specified domains.

Key Insights and Ideas

  • Simultaneous Multi-Domain Service is Achievable: The most significant insight is that dialog systems can effectively serve multiple domains concurrently, moving beyond the single-domain limitation of prior work.
  • Hierarchical Domain Models are Key: The HDM provides a structured way to represent domain knowledge and relationships, which is fundamental for accurate multi-domain selection.
  • Context and History are Crucial: The framework's ability to consider dialog history and context in final domain determination is vital for disambiguating user intent and avoiding unnecessary prompts.
  • Automatic Annotation for Overlapping Domains: The development of an automatic multi-domain label annotation process is a critical step for creating training data that accurately reflects the complexities of utterances belonging to multiple related domains.

Key Results

  • Candidate Domain Detection: The proposed HDM-based MDSF achieved significantly higher recall (96.2%) and F-1 score (95.9%) compared to a baseline (65.2% recall, 78.0% F-1 score). While precision was slightly lower (95.6% vs. 97.1%), the substantial increase in recall indicates a much better ability to identify all relevant domains, especially when an utterance could apply to multiple.
  • Multi-Domain Dialog System Performance: In human user experiments, the system employing the HDM-based MDSF showed marked improvements:
  • Successful Turn Rate (STR): Increased from 55.0% (baseline) to 91.1% (proposed).
  • Task Completion Rate (TCR): Increased from 58.8% (baseline) to 95.0% (proposed).
  • Average Turn Length (ATL): Decreased from 4.7 (baseline) to 3.5 (proposed), indicating more efficient interactions.

Limitations and Future Work

  • Limitations: The paper acknowledges that the recall of the baseline candidate domain detection component was low due to numerous false negatives, particularly when utterances could be interpreted by multiple domains. The proposed system, while improving recall, had slightly lower precision.
  • Future Work: The authors plan to research multi-domain user simulation. They highlight that human user experiments are time-consuming and expensive, and a robust user simulator capable of mimicking users in multi-domain dialog systems employing MDSF is needed for more efficient evaluation. This multi-domain user simulation is identified as an important area for future research in domain selection.

Summary updated

BibTeX
@inproceedings{ryu2012hierarchical,
  author = {Seonghan Ryu and Donghyeon Lee and Injae Lee and Sangdo Han and Gary Geunbae Lee and Myungjae Kim and Kyungduk Kim},
  title = {{A Hierarchical Domain Model-Based Multi-Domain Selection Framework for Multi-Domain Dialog Systems}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
}
Figure