Learning From Unannotated QA Pairs to Analogically Disambiguate and Answer Questions

Maxwell Crouse, Clifton James McFate, Kenneth D. ForbusAAAI 2018 · DOI

Introduction

Building Natural Language Question Answering (NLQA) systems typically follows one of two paths: supervised learning, which requires labor-intensive manual mapping of questions to logical forms, or weakly supervised learning from question-answer pairs, which often suffers from poor data efficiency and brittleness. This paper addresses the "cold start" problem: adapting a robust, general-purpose language system to a specific new domain (like US Geography) without requiring a mountain of hand-labeled data.

Methods

The researchers from Northwestern University’s Qualitative Reasoning Group propose an analogy-based approach building upon the ResearchCyc ontology and the EA NLU semantic parser. The technique consists of two main phases: training and inference.

Training Phase: Learning from QA Pairs

When given a question (e.g., "What states border Indiana?") and its answers (Illinois, Michigan, etc.), the system performs pathfinding in the Cyc knowledge base to connect entities. It uses a weighted set-cover algorithm to construct the simplest combination of paths explaining the answer set. These successful queries are deconstructed into Evidence Expressions (EEs)—logic snippets linking linguistic patterns to database predicates. Ambiguities, such as whether "state" refers to a geopolitical entity or a state of matter, are resolved via hypergraph partitioning across the case library.

Inference Phase: Answering New Questions

To answer a new question, such as "What states border states that border Colorado?", the system employs analogical retrieval. It uses the MAC/FAC (Many Are Called/Few Are Chosen) algorithm to find similar previously learned cases.

The inference process uses analogical retrieval and composition to build queries for new questions.

The system does not simply look for whole-sentence matches; it retrieves relevant EEs and "stitches" them together compositionally. The Structure Mapping Engine (SME) aligns the structure of the new question with stored cases to form a brand-new query.

Datasets and Evaluation

The system was evaluated on Geoquery, a benchmark of 880 US geography questions. The authors identified 22 incorrect "gold standard" answers in the original dataset and provided results for both "Uncorrected" and "Corrected" versions for a fairer assessment.

Key Results

The system achieved state-of-the-art performance, reaching 88.8% accuracy on the corrected Geoquery dataset using 10-fold cross-validation. This performance is comparable to systems using manually annotated logical forms.

The system demonstrates extreme data efficiency, reaching over 80% accuracy with only 60 training examples.

A standout feature is the system's extreme data efficiency. As shown in the learning curve, it achieves approximately 51.6% accuracy with just 10 examples and exceeds 80% accuracy with only 60 examples. Furthermore, the model generalizes well, with minimal performance drops when moving from 10-fold to 2-fold cross-validation (88.8% to 87.4%).

Insights and Discussion

The success of this approach stems from generalization over specialization. By starting with a system that understands basic English syntax, the model focuses purely on learning the mapping to domain-specific facts. Structural analogy allows for faster rule learning compared to pure statistical correlation. Additionally, the system can "abduce" (infer) missing links using its knowledge base and handle complex, nested questions by combining fragments learned from simpler ones.

Current limitations include a dependency on pre-existing formal knowledge bases like Cyc and a focus on "wh-questions." Future work aims to expand to more linguistically diverse datasets and explore learning from non-exhaustive answer lists. Because the system utilizes symbolic logic and analogy, it offers significant potential for explainable AI, providing justifications for its answers based on retrieved cases.

Summary updated

BibTeX
@inproceedings{crouse2018learning,
  author = {Maxwell Crouse and Clifton James McFate and Kenneth D. Forbus},
  title = {{Learning From Unannotated QA Pairs to Analogically Disambiguate and Answer Questions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11329},
}
Figure