Open Dialogue Management for Relational Databases

Ben Hixon, Rebecca J. PassonneauNAACL 2013

TL;DR

This paper introduces ODDMER, a novel system for "open dialogue management" that automatically learns how to interact with users to query relational databases. By analyzing database semantics, ODDMER can infer user vocabulary, identify relevant tables (dialogue foci), and generate efficient dialogue strategies, significantly improving the experience for users with limited domain knowledge.

What Problem Does This Paper Address?

Traditional dialogue systems for relational databases often struggle with the "vocabulary problem" – the mismatch between the system's internal terminology and the user's everyday language. This necessitates either extensive manual pre-processing to define user-understandable attributes or handcrafted dialogue strategies, limiting portability and scalability. Furthermore, these systems often assume users have complete knowledge of the database schema and its contents, leading to inefficient or failed interactions for novices. This paper aims to bridge this gap by creating a dialogue manager that can adapt to a user's likely vocabulary and intelligently select relevant parts of the database to focus on, making database interaction more accessible.

Methods

ODDMER employs a three-pronged approach to achieve open dialogue management:

  • Vocabulary Selection: This module uses supervised learning to identify "intelligible attributes" within each database table. These are attributes whose values are likely to be understood by a typical user, effectively filtering out jargon or internal codes. This is achieved by training a binary classifier on labeled attributes from a benchmark database.
  • Focus Discovery: This module identifies "candidate dialogue foci" – tables that are most likely to address basic user goals. It uses a "schema summarization" technique, inspired by graph-based algorithms, to rank tables based on their size, the entropy of their intelligible attributes, and their connectivity to other tables (joins). This process prioritizes tables that are both semantically rich and relevant to user queries.
  • Focus Agent Generation: For each identified dialogue focus, a "focus agent" is instantiated. This agent then generates a default dialogue strategy that elicits attribute values from the user in order of their "semantic specificity." Semantic specificity measures how uniquely an attribute's values map to rows in the database, allowing the system to ask for the most distinguishing information first.

Datasets

  • Training Data: A set of 84 attributes from the Microsoft AdventureWorks Cycle Company database was used to train the vocabulary selection classifier.
  • Evaluation Data: The system was evaluated on three distinct databases:
  • Heiskell Library: A library database with tables like "BOOK" and "CIRCULATION HISTORY."
    • Grocery: A small supermarket database.
  • Eve Online: A database for a massively multiplayer online role-playing game, notable for its primarily numeric data.

Key Insights and Ideas

  • Database Semantics as a Foundation: The core idea is that the structure and content of the database itself hold the key to understanding user interaction, rather than relying solely on external knowledge or manual annotation.
  • Intelligibility Over All Attributes: By focusing on intelligible attributes rather than all attributes, ODDMER can significantly reduce the vocabulary gap and improve user experience.
  • Prioritizing "Basic" Goals: The focus discovery mechanism intelligently identifies the most relevant tables for common user goals, streamlining the interaction by not overwhelming the user with the entire database schema.
  • Specificity as an Efficiency Driver: Ordering prompts by semantic specificity, even without full vocabulary knowledge, dramatically improves dialogue efficiency by guiding users to provide the most useful information first.

Key Results

  • Improved Dialogue Efficiency for Novices: Simulated users with incomplete domain knowledge, using ODDMER's vocabulary selection and specificity-ordered prompts, achieved dialogue lengths nearly as efficient as users with complete domain knowledge. For example, in the Heiskell domain, incomplete-knowledge users saw their dialogue length decrease from an average of 25.3 turns (no vocabulary selection, random order) to 16.5 turns (no vocabulary selection, specificity order), and further improved with vocabulary selection.
  • Portability Demonstrated: ODDMER was successfully applied to three very different databases, showcasing its domain-independent nature.
  • Vocabulary Selection Effectiveness: Vocabulary selection was found to be particularly effective for users with incomplete knowledge, even when prompts were in a random order.

Limitations and Future Work

  • Numeric Databases and Ambiguous Goals: The paper notes that focus discovery and vocabulary selection are less effective on purely numeric databases where dialogue goals might be less clear or less tied to semantic richness (e.g., the Eve Online database).
  • Table and Attribute Labeling: ODDMER currently relies on database designer-provided labels. Future work aims to infer more meaningful labels by consulting external knowledge bases.
  • Beyond Tuple Selection: The current system primarily focuses on tuple selection. Future work will explore extending focus discovery and vocabulary selection to handle more complex goals like information aggregation or finding alternative items.
  • Real-World Evaluation: The current evaluation uses simulated users. The authors express a desire to conduct evaluations with real users in the future.

Summary updated

BibTeX
@inproceedings{hixon2013open,
  author = {Ben Hixon and Rebecca J. Passonneau},
  title = {{Open Dialogue Management for Relational Databases}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure