Hybrid Question Answering over Knowledge Base and Free Text

Kun Xu, Yansong Feng, Songfang Huang, Dongyan ZhaoCOLING 2016 · OpenReview

Introduction

Modern Question Answering (QA) systems often face a "completeness gap." Structured Knowledge Bases (KBs) like DBpedia offer high precision but suffer from limited coverage, often referred to as the "closed vocabulary" problem. Conversely, free-text corpora like Wikipedia contain vast amounts of human knowledge but are difficult to query for complex, multi-hop questions. This research, presented at COLING 2016, proposes a hybrid framework that bridges this gap by allowing a QA engine to perform joint inference across both structured predicates and natural language relations simultaneously.

Methodology

The core of the proposed system is a three-stage pipeline designed to integrate heterogeneous data sources into a cohesive inference process.

  1. Question Decomposition: Complex natural language questions are broken down into "ungrounded triples." For example, the question "Who is the front man of the band that wrote Coffee & TV?" is decomposed into triples such as <ans, is the front man of, var1> and <var1, wrote, Coffee & TV>.
  2. Candidate Generation:
    • Entity Linking: Potential KB entities are identified using tools like DBpedia Lookup.
  3. Hybrid Relation Extraction: For every relational phrase, the system predicts both KB predicates (via a Multi-Channel Convolutional Neural Network) and textual relations (via a paraphrase model based on Recursive Autoencoders).
  4. Joint Inference (ILP): To ensure global coherence, the system employs Integer Linear Programming (ILP). This step finds a globally optimal solution by enforcing constraints such as entity type compatibility (e.g., ensuring a "Band" is linked to a "front man" relation) and semantic alignment between KB predicates and textual relations.
Figure 1: The hybrid QA pipeline illustrating question decomposition, candidate generation from KB and text, and joint inference via ILP.

Experimental Setup

The system was evaluated on two primary benchmarks:

  • WebQuestions: A standard dataset of 5,810 QA pairs, primarily answerable via Freebase.
  • QALD-6 (Hybrid Task): A challenging dataset specifically requiring information from both DBpedia and free text to be answered correctly.

Results and Discussion

The hybrid approach demonstrated significant improvements over single-source baselines. On the WebQuestions dataset, the system achieved a state-of-the-art F1 score of 53.8. More importantly, on the QALD-6 hybrid task, the full "KB + Text + Joint" model reached an F1 of 40.9, vastly outperforming models restricted to only KB (10.1 F1) or only Text (28.7 F1).

Table: Performance comparison of the hybrid model against state-of-the-art methods on WebQuestions and an ablation study on QALD-6.

Key insights from the evaluation include:

  • Semantic Coherence: Using KB entity types to filter textual relations effectively removes logically impossible candidates.
  • Redundancy as Verification: Mapping phrases to both sources allows the system to use one to verify the other, increasing confidence when both point to the same answer.
  • Joint Optimization: The ILP step provided a consistent performance boost of at least 3%, highlighting the importance of global optimization over simple pipelined approaches.

Conclusion

While the hybrid framework sets a new benchmark for QA, limitations remain. The system currently struggles with logical aggregations (e.g., "second longest river") and very short, ambiguous relational phrases. Future work aims to develop shallow semantic parsers that can better capture natural language nuances while maintaining the flexibility of the hybrid architecture.

Summary updated

BibTeX
@inproceedings{xu2016hybrid,
  author = {Kun Xu and Yansong Feng and Songfang Huang and Dongyan Zhao},
  title = {{Hybrid Question Answering over Knowledge Base and Free Text}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
  url = {https://openreview.net/forum?id=N7EBAe9DPQ},
}
Figure