Learning Knowledge Graphs for Question Answering through Conversational Dialog

Ben Hixon, Peter Clark, Hannaneh Hajishirzi

Introduction

Advanced Question Answering (QA) systems often struggle with "missing links" in their knowledge bases. For instance, a system might know that "metal conducts electricity," but fail to answer a question about an iron nail because it lacks the specific fact that "iron is a metal." Traditionally, bridging these gaps required extensive manual engineering or rigid ontologies.

This paper introduces KNOWBOT, a system designed for dynamic knowledge acquisition. KNOWBOT learns missing common-sense relations through natural, unscripted conversations with users, extracting "bridge" relations and storing them in a Knowledge Graph (KG) to improve its reasoning capabilities.

An example dialog where KNOWBOT learns the relationship between iron, metal, and electricity to solve a science question.

Methodology

KNOWBOT constructs a Knowledge Graph through a three-stage hierarchical process:

  1. Utterance-level (uKG): When a user provides an explanation (e.g., "the heat from the stove causes the snow to melt"), the system generates a fully connected graph of keywords. To filter natural language noise, it applies two constraints:
  2. Alignment: Only edges connecting a concept in the question to a concept in the supporting fact corpus are retained.
  3. Adjacency: Edges between words appearing consecutively in the sentence are pruned, as these are often "filler" words.
  4. Dialog-level (dKG): Successful extractions from individual utterances are aggregated into a graph specific to the current question.
  5. Global-level (gKG): After a dialog concludes, relations are pooled into a master graph. A "redundancy filter" ensures quality by ignoring relations mentioned only once across all users.
The construction of utterance-level (uKG) and dialog-level (dKG) knowledge graphs from user explanations.

The system employs two dialog strategies: User-initiative, where it asks open-ended "Why?" questions, and Mixed-initiative, where it asks targeted questions about specific concepts it identifies as important.

Experimental Setup

The researchers evaluated KNOWBOT using:

  • Questions: 107 science questions from the 4th-grade New York Regents exam.
  • Fact Corpus (SCITEXT): A collection of unlabeled true/false sentences from science textbooks and Wikipedia.
  • Dialog Data: A novel dataset of real user dialogs where participants corrected the system and provided explanations.

Key Insights

  • Task-Driven vs. General Knowledge: Relations learned from brief user interactions proved more effective for solving specific science questions than massive, general-purpose databases like WordNet or the Paraphrase Database (PPDB), which often introduce irrelevant noise.
  • Viability of Open Dialog: Unlike virtual assistants restricted to closed models, KNOWBOT operates in an open domain, using the task goal to interpret free-text responses.
  • Effective Pruning: Simple syntactic rules, such as the Adjacency constraint, allow the system to extract meaningful semantic relations from ungrammatical human speech without a complex semantic parser.
  • Mixed-Initiative Efficiency: Users preferred the mixed-initiative strategy, which was more efficient and required fewer "bridge" relations to reach the correct answer.

Results

KNOWBOT significantly outperformed baseline systems:

  • QA Accuracy: KNOWBOT achieved 57% accuracy, compared to a 41% baseline using only keyword matching. It also surpassed WordNet (34%) and PPDB (39%).
  • Dialog Success: Users successfully guided KNOWBOT to the correct answer in approximately 50% of dialogs, whereas a "Query Expansion" baseline succeeded only 5% of the time.

Limitations and Future Work

Despite its success, KNOWBOT has limitations. Its reasoning is currently based on overlap scores, which can fail to distinguish directional relationships (e.g., "liquid to solid" vs. "solid to liquid"). It also struggles with multi-step logic or mathematical questions. Future work includes integrating dependency parsers for better syntactic understanding and using reinforcement learning to optimize dialog strategies.

The Global Knowledge Graph showed only slight knowledge transfer between questions, suggesting that a larger dataset is necessary to fully leverage the power of aggregated user knowledge.

Summary updated

Abstract

We describe how a question-answering system can learn about its domain from conversational dialogs. Our system learns to relate concepts in science questions to propositions in a fact corpus, stores new concepts and relations in a knowledge graph (KG), and uses the graph to solve questions. We are the first to acquire knowledge for question-answering from open, natural language dialogs without a fixed ontology or domain model that predetermines what users can say. Our relation-based strategies complete more successful dialogs than a query expansion baseline, our taskdriven relations are more effective for solving science questions than relations from general knowledge sources, and our method is practical enough to generalize to other domains.
BibTeX
@inproceedings{hixon2015learning,
  author = {Ben Hixon and Peter Clark and Hannaneh Hajishirzi},
  title = {{Learning Knowledge Graphs for Question Answering through Conversational Dialog}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1086},
}
Figure