“Dialog Navigator”: A Question Answering System Based on Large Text Knowledge Base

Yoji Kiyota, Sadao Kurohashi, Fuyuko KidoCOLING 2002

TL;DR

This paper introduces "Dialog Navigator," a question answering (QA) system designed to handle vague user queries by engaging in a clarifying dialogue. It tackles the challenge of imprecise user questions in large text knowledge bases by employing precise text retrieval, synonymous expression handling, and a novel dialog management system using "dialog cards" to guide users to their answers.

What Problem Does This Paper Address?

Traditional question answering systems often struggle with the inherent vagueness of natural language queries. Users rarely pose perfectly specific questions, leading to systems returning overwhelming lists of irrelevant results or failing to find answers altogether. This is particularly problematic when dealing with large, unstructured text knowledge bases, where the gap between a user's intent and the precise wording required for retrieval can be significant. The authors highlight that a substantial portion (around 25%) of user queries are vague, necessitating a more interactive approach than simple one-step retrieval. The core problem is bridging the semantic gap between user questions and the information contained within a large knowledge base, especially when the user cannot articulate their need precisely.

Methods

The "Dialog Navigator" system employs several key techniques to address the problem of vague queries:

  • Precise Text Retrieval:
    • Question Type Classification: The system first classifies user questions into types like "symptom," "how," or "what." This allows for targeted retrieval from specific text collections (e.g., Glossary for "what" questions, Help texts for "how" questions).
  • Synonymous Expression Dictionary: To overcome expression gaps, the system utilizes a dictionary of synonyms and synonymous phrases. This dictionary supports recursive relations and is expanded to cover a wide range of expressions, enabling the system to match variations in user phrasing.
  • Modifier-Head Relation Matching: The system gives higher weight to matching modifier-head relations (e.g., "open → file") to improve retrieval precision.
  • Indexing: An inverted index of keywords and synonymous expressions is used for fast retrieval.
  • Dialog Management and User Navigation:
    • Dialog Cards: For vague questions, the system uses "dialog cards" which are pre-defined conversational flows. If a user's question matches a card, the system asks clarifying questions with multiple-choice options. These options can either lead to further dialog cards (e.g., "goto") or trigger a refined search query (e.g., "retrieve").
  • Description Extraction: After retrieving relevant texts, the system extracts concise descriptions from the neighborhood of the matching parts. This helps users quickly understand the relevance of a retrieved document without having to read the entire text. For certain collections like Support KB, titles are used as descriptions.

Datasets

The paper utilizes the following text collections for its knowledge base:

  • Glossary: 4,707 texts.
  • Help texts: 11,306 texts.
  • Support KB (Microsoft Support Knowledge Base): 23,323 texts.

The evaluation was performed on a dataset of 603 user sessions collected from the system's live service.

Key Insights and Ideas

  • The necessity of dialogue for vague queries: The paper strongly argues that for a significant portion of user queries, a one-shot retrieval approach is insufficient. A dialog system is crucial for clarifying intent and guiding users to the correct information.
  • Leveraging existing knowledge bases: Instead of building a knowledge base from scratch, the system effectively utilizes existing, large-scale knowledge bases like Microsoft Support KB. This significantly reduces the cost and effort of system construction.
  • Synonymy as a key to bridging expression gaps: The comprehensive synonymous expression dictionary, including recursive relations, is a powerful tool for matching user queries to text content, even when phrasing differs.
  • "Dialog cards" as an intuitive navigation tool: The concept of dialog cards provides a structured and user-friendly way to handle ambiguity. By offering clear choices, the system effectively narrows down the search space and guides the user.
  • Description extraction for concise answers: Extracting relevant snippets rather than just returning entire documents significantly improves user experience by highlighting the most pertinent information.

Key Results

  • Success Rate: The system achieved a success ratio of 52% (or 66% if "Failure A" - no relevant texts found - is considered a success).
  • Dialog Card Impact: Dialog cards were used in 12 sessions, all of which were successful. The authors estimate an improvement of +3.8% in the success ratio due to dialog cards.
  • Description Extraction Quality: 61% of extracted descriptions were judged as "Proper." When considering titles from Glossary and Help texts as proper descriptions, this figure rose to 73%.

Limitations and Future Work

  • System Domain: The system is primarily designed for Microsoft product-related queries, limiting its applicability to other domains without significant adaptation.
  • Knowledge Base Scale: While the system uses large text knowledge bases, the authors acknowledge that the amount of knowledge is still a limiting factor for QA systems in general.
  • Manual Construction of Dialog Cards: The dialog cards are manually created, which could be a bottleneck for expanding the system to new domains or handling a wider variety of vague questions. Future work could explore automated methods for generating or expanding these dialog cards.
  • Cost of Knowledge Base: Although the system utilizes existing knowledge bases, the initial construction and maintenance of these bases still represent a cost.

The paper suggests that further research could focus on improving the scale and coverage of the knowledge base and potentially automating the creation of dialog structures.

Summary updated

BibTeX
@inproceedings{kiyota2002dialog,
  author = {Yoji Kiyota and Sadao Kurohashi and Fuyuko Kido},
  title = {{“Dialog Navigator”: A Question Answering System Based on Large Text Knowledge Base}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2002},
}
Figure