Introduction
Advanced Question Answering (QA) systems often struggle with "missing links" in their knowledge bases. For instance, a system might know that "metal conducts electricity," but fail to answer a question about an iron nail because it lacks the specific fact that "iron is a metal." Traditionally, bridging these gaps required extensive manual engineering or rigid ontologies.
This paper introduces KNOWBOT, a system designed for dynamic knowledge acquisition. KNOWBOT learns missing common-sense relations through natural, unscripted conversations with users, extracting "bridge" relations and storing them in a Knowledge Graph (KG) to improve its reasoning capabilities.
Methodology
KNOWBOT constructs a Knowledge Graph through a three-stage hierarchical process:
- Utterance-level (uKG): When a user provides an explanation (e.g., "the heat from the stove causes the snow to melt"), the system generates a fully connected graph of keywords. To filter natural language noise, it applies two constraints:
- Alignment: Only edges connecting a concept in the question to a concept in the supporting fact corpus are retained.
- Adjacency: Edges between words appearing consecutively in the sentence are pruned, as these are often "filler" words.
- Dialog-level (dKG): Successful extractions from individual utterances are aggregated into a graph specific to the current question.
- Global-level (gKG): After a dialog concludes, relations are pooled into a master graph. A "redundancy filter" ensures quality by ignoring relations mentioned only once across all users.
The system employs two dialog strategies: User-initiative, where it asks open-ended "Why?" questions, and Mixed-initiative, where it asks targeted questions about specific concepts it identifies as important.
Experimental Setup
The researchers evaluated KNOWBOT using:
- Questions: 107 science questions from the 4th-grade New York Regents exam.
- Fact Corpus (SCITEXT): A collection of unlabeled true/false sentences from science textbooks and Wikipedia.
- Dialog Data: A novel dataset of real user dialogs where participants corrected the system and provided explanations.
Key Insights
- Task-Driven vs. General Knowledge: Relations learned from brief user interactions proved more effective for solving specific science questions than massive, general-purpose databases like WordNet or the Paraphrase Database (PPDB), which often introduce irrelevant noise.
- Viability of Open Dialog: Unlike virtual assistants restricted to closed models, KNOWBOT operates in an open domain, using the task goal to interpret free-text responses.
- Effective Pruning: Simple syntactic rules, such as the Adjacency constraint, allow the system to extract meaningful semantic relations from ungrammatical human speech without a complex semantic parser.
- Mixed-Initiative Efficiency: Users preferred the mixed-initiative strategy, which was more efficient and required fewer "bridge" relations to reach the correct answer.
Results
KNOWBOT significantly outperformed baseline systems:
- QA Accuracy: KNOWBOT achieved 57% accuracy, compared to a 41% baseline using only keyword matching. It also surpassed WordNet (34%) and PPDB (39%).
- Dialog Success: Users successfully guided KNOWBOT to the correct answer in approximately 50% of dialogs, whereas a "Query Expansion" baseline succeeded only 5% of the time.
Limitations and Future Work
Despite its success, KNOWBOT has limitations. Its reasoning is currently based on overlap scores, which can fail to distinguish directional relationships (e.g., "liquid to solid" vs. "solid to liquid"). It also struggles with multi-step logic or mathematical questions. Future work includes integrating dependency parsers for better syntactic understanding and using reinforcement learning to optimize dialog strategies.
The Global Knowledge Graph showed only slight knowledge transfer between questions, suggesting that a larger dataset is necessary to fully leverage the power of aggregated user knowledge.
Summary updated