Exploiting Rich Syntax for Better Knowledge Base Question Answering

Pengju Zhang, Yonghui Jia, Muhua Zhu, Wenliang Chen, Min ZhangarXiv 2021 · arXiv

Abstract

Knowledge Base Question Answering (KBQA) systems often struggle to transform complex natural language into precise database queries due to a reliance on simple word sequences. This paper introduces a syntax-aware framework that incorporates grammatical structure through Shortest Dependency Paths (SDP), Tree Position Features (TPF), and Tree-structured Gated Recurrent Units (Tree-GRU). By integrating these path-based and tree-based representations, the model effectively captures long-distance dependencies that sequential models often miss. Experimental results on the WebQuestions benchmark demonstrate that explicit syntactic features significantly improve performance, achieving an F1 score of 53.63% and outperforming sequential baselines, particularly on longer and more complex queries.

Introduction: The Syntax Bottleneck

The core challenge in KBQA is question understanding—mapping a messy human sentence like "What movies did Diana play in?" to a structured query. Most existing models rely on sequential representations, which can fail to capture critical long-distance grammatical connections. For instance, in the question shown below, the relationship between "movies" and "play" is central to the query but is separated by other words in the linear sequence.

Dependency tree illustrating grammatical relationships in a question.

Syntax provides a "shortcut" for these connections. By utilizing dependency trees, models can directly link related words regardless of their linear distance, leading to more accurate semantic interpretations and more precise database queries.

Methodology: Integrating Path and Tree Representations

The authors propose a multi-faceted approach to capture syntax, combining three distinct representations into a unified model:

  1. Shortest Dependency Path (SDP): A path-based method that identifies the grammatical links between the "focus word" (entity) and the "answer word" (wh-word), filtering out irrelevant linguistic noise.
  2. Tree Position Feature (TPF): A tree-based method that encodes the hierarchical distance of each word from the root of the dependency tree, providing global structural context.
  3. Tree-GRU (Tree-structured Gated Recurrent Unit): A neural network architecture designed to process the entire dependency tree. The model is pre-trained on nearly 5 million questions to learn the nuances of dependency edges.
The overall architecture of the syntax-aware KBQA model, integrating SDP, TPF, and Tree-GRU components.

These features are accumulated with standard word-sequence representations (BiGRU) and powerful embeddings like ELMo to create a robust, syntax-aware question encoding. This combined representation is then used to calculate similarity against potential knowledge base paths.

Experimental Setup and Results

The model was evaluated on the WebQuestions benchmark using Freebase as the knowledge source. To support the Tree-GRU, a large-scale pre-training corpus of 4.99 million questions was curated from sources like SimpleQuestions and Quora.

Key Results

  • State-of-the-Art Accuracy: The combined model achieved an F1 score of 53.63% on WebQuestions, surpassing previous high-performing baselines.
  • Complementary Features: Ablation studies showed that SDP and TPF provide unique, non-redundant information. Combining them yielded better results than using either alone, suggesting they capture different aspects of grammatical structure.
  • Performance on Long Questions: The most significant gains were observed in complex, long-form questions where sequential models typically fail to maintain structural coherence. Syntactic features act as a bridge, allowing the model to "jump" across the sentence to find relevant constraints.

Insights and Future Directions

The research highlights that even advanced language models like ELMo do not fully capture explicit syntax for specialized tasks like KBQA. Explicitly modeling the underlying "tree" of a sentence is essential for maintaining accuracy as questions grow in complexity.

Future work may involve integrating these syntactic representations into more advanced reasoning architectures, such as Fact Memory Networks, to handle multi-hop questions more effectively. While the Tree-GRU requires significant pre-training, its ability to model the formal structure of grammar proves essential for high-precision question answering.

Summary updated

Abstract

Recent studies on Knowledge Base Question Answering (KBQA) have shown great progress on this task via better question understanding. Previous works for encoding questions mainly focus on the word sequences, but seldom consider the information from syntactic trees.In this paper, we propose an approach to learn syntax-based representations for KBQA. First, we encode path-based syntax by considering the shortest dependency paths between keywords. Then, we propose two encoding strategies to mode the information of whole syntactic trees to obtain tree-based syntax. Finally, we combine both path-based and tree-based syntax representations for KBQA. We conduct extensive experiments on a widely used benchmark dataset and the experimental results show that our syntax-aware systems can make full use of syntax information in different settings and achieve state-of-the-art performance of KBQA.
BibTeX
@misc{zhang2021exploiting,
  author = {Pengju Zhang and Yonghui Jia and Muhua Zhu and Wenliang Chen and Min Zhang},
  title = {{Exploiting Rich Syntax for Better Knowledge Base Question Answering}},
  year = {2021},
  eprint = {2107.07940},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.07940},
}
Figure