Introduction
Modern Question Answering (QA) systems often face a "completeness gap." Structured Knowledge Bases (KBs) like DBpedia offer high precision but suffer from limited coverage, often referred to as the "closed vocabulary" problem. Conversely, free-text corpora like Wikipedia contain vast amounts of human knowledge but are difficult to query for complex, multi-hop questions. This research, presented at COLING 2016, proposes a hybrid framework that bridges this gap by allowing a QA engine to perform joint inference across both structured predicates and natural language relations simultaneously.
Methodology
The core of the proposed system is a three-stage pipeline designed to integrate heterogeneous data sources into a cohesive inference process.
- Question Decomposition: Complex natural language questions are broken down into "ungrounded triples." For example, the question "Who is the front man of the band that wrote Coffee & TV?" is decomposed into triples such as
<ans, is the front man of, var1>and<var1, wrote, Coffee & TV>. - Candidate Generation:
- Entity Linking: Potential KB entities are identified using tools like DBpedia Lookup.
- Hybrid Relation Extraction: For every relational phrase, the system predicts both KB predicates (via a Multi-Channel Convolutional Neural Network) and textual relations (via a paraphrase model based on Recursive Autoencoders).
- Joint Inference (ILP): To ensure global coherence, the system employs Integer Linear Programming (ILP). This step finds a globally optimal solution by enforcing constraints such as entity type compatibility (e.g., ensuring a "Band" is linked to a "front man" relation) and semantic alignment between KB predicates and textual relations.
Experimental Setup
The system was evaluated on two primary benchmarks:
- WebQuestions: A standard dataset of 5,810 QA pairs, primarily answerable via Freebase.
- QALD-6 (Hybrid Task): A challenging dataset specifically requiring information from both DBpedia and free text to be answered correctly.
Results and Discussion
The hybrid approach demonstrated significant improvements over single-source baselines. On the WebQuestions dataset, the system achieved a state-of-the-art F1 score of 53.8. More importantly, on the QALD-6 hybrid task, the full "KB + Text + Joint" model reached an F1 of 40.9, vastly outperforming models restricted to only KB (10.1 F1) or only Text (28.7 F1).
Key insights from the evaluation include:
- Semantic Coherence: Using KB entity types to filter textual relations effectively removes logically impossible candidates.
- Redundancy as Verification: Mapping phrases to both sources allows the system to use one to verify the other, increasing confidence when both point to the same answer.
- Joint Optimization: The ILP step provided a consistent performance boost of at least 3%, highlighting the importance of global optimization over simple pipelined approaches.
Conclusion
While the hybrid framework sets a new benchmark for QA, limitations remain. The system currently struggles with logical aggregations (e.g., "second longest river") and very short, ambiguous relational phrases. Future work aims to develop shallow semantic parsers that can better capture natural language nuances while maintaining the flexibility of the hybrid architecture.
Summary updated