KBQA: An Online Template Based Question Answering System over Freebase

Wanyun Cui, Yanghua Xiao, Wei Wang

Abstract

KBQA is a system designed to bridge the gap between natural language questions and structured knowledge bases like Freebase. By learning over 1.1 million question templates from massive web-scale QA corpora (Yahoo! Answers), it can accurately map diverse human queries to specific database predicates, enabling precise answers to both simple and complex factoid questions. This paper details the two-phase architecture—offline template extraction and online question answering—and demonstrates how the system achieves an order of magnitude higher predicate coverage compared to traditional bootstrapping methods.

Introduction

Knowledge Bases (KBs) like Freebase contain billions of facts, but querying them usually requires specialized languages like SPARQL. For a regular user to ask a simple question like "How many people live in Honolulu?", a system must perform semantic parsing—translating that natural language into a structured query like (Honolulu, location.population, ?x).

The core challenge is the semantic gap: there are infinite ways to ask the same question (e.g., "What is the population of $city?" vs. "How many people are there in $city?"). Previous methods often relied on manual rules or small-scale "bootstrapping" techniques that lacked the coverage to handle the sheer variety of human expression and the thousands of different relations stored in a modern KB.

Methodology

The authors propose a template-based approach that operates in two distinct phases: offline extraction and online answering.

The system architecture of KBQA, illustrating the interaction between offline template extraction and online query processing.

1. Offline Template Extraction

Instead of hand-coding rules, KBQA "learns" from 41 million QA pairs from Yahoo! Answers. The system identifies entities in the question and their corresponding values in the answer, then looks up the relationship (predicate) between them in Freebase. To ensure generality, specific entities are replaced with concepts (e.g., "Honolulu" becomes $city) using Probase, a massive semantic network.

2. Online Question Answering

When a user submits a question, the system identifies the entity using Stanford NER and replaces it with its concept. It then matches the resulting template against its repository of 1.1 million learned patterns. Finally, it retrieves the answer from Freebase, handling complex KB structures where a relationship might be represented by a path (e.g., spouse -> name) rather than a single link.

Datasets

  • Yahoo! Answers: A corpus of 41 million QA pairs used to harvest templates.
  • Freebase: The target large-scale knowledge base containing the facts.
  • Probase: A semantic network used for conceptualization (mapping entities to categories like city, author, or athlete).
  • Comparison Baseline: The system was compared against BOA (Bootstrapping Ontology Alignment) patterns, derived from a corpus of 256 million sentences.

Key Results

The evaluation of KBQA demonstrates the power of scale in template learning. KBQA discovered over 4,600 distinct relations, an order of magnitude higher than the 283 identified by the BOA baseline.

A comparison of corpus size and resulting templates/predicates between KBQA and the bootstrapping baseline.

Despite using a smaller initial corpus (41M QA pairs vs. 256M sentences for BOA), KBQA found significantly more templates and predicates. This suggests that QA pairs are a much "richer" source of training data for this task than raw sentences. Furthermore, the system successfully demonstrated the ability to decompose complex questions into multiple "Binary Factoid Questions" (BFQs) to retrieve multi-part answers.

Conclusion and Future Work

KBQA demonstrates that templates serve as an effective middle ground between rigid rules and unstructured search. By leveraging massive QA corpora and conceptualization via Probase, the system achieves high coverage and precision. However, the system's accuracy remains reliant on the performance of external tools like Stanford NER. Future work will focus on addressing the ambiguity inherent in certain question templates and extending the system's capabilities to handle more complex multi-hop reasoning or temporal questions.

Summary updated

Abstract

Question answering (QA) has become a popular way for humans to access billion-scale knowledge bases. QA systems over knowledge bases produce accurate and concise answers. The key of QA over knowledge bases is to map the question to a certain substructure in the knowledge base. To do this, KBQA (Question Answering over Knowledge Bases) uses a new kind of question representation: templates, learned from a million scale QA corpora. For example, for questions about a city's population, KBQA learns templates such as What's the population of city?, How many people are there in city?. It learns overall 1171303 templates for 4690 relations. Based on these templates, KBQA effectively and efficiently supports binary factoid questions or complex questions.
BibTeX
@inproceedings{cui2016kbqa,
  author = {Wanyun Cui and Yanghua Xiao and Wei Wang},
  title = {{KBQA: An Online Template Based Question Answering System over Freebase}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure