Crake: Causal-Enhanced Table-Filler for Question Answering over Large Scale Knowledge Base

Minhao Zhang, Ruoyu Zhang, Yanzeng Li, Lei Zou

Abstract

Knowledge Base Question Answering (KBQA) requires converting natural language into structured queries like SPARQL. Traditional methods often treat this as a sequence-generation task, which suffers from exposure bias and ignores the inherent causal dependencies between node identification and graph construction. We present Crake, a framework that utilizes a causal-enhanced table-filling approach for Graph Structure Generation (GSG) and an efficient beam-search for Relation Extraction (RE). By modeling graph construction as a set-based prediction task rather than a sequence, Crake achieves a significant performance leap, reaching an F1-score of 0.715 on the LC-QuAD 1.0 benchmark.

1. Introduction

Answering complex questions using large-scale Knowledge Bases (KBs) such as DBpedia involves a multi-step semantic parsing pipeline: Node Extraction (NE), Graph Composition (GC), and Relation Extraction (RE). Current state-of-the-art models typically treat these steps as independent sequence-to-sequence tasks. However, this approach faces two primary hurdles:

  1. Causal Neglect: There is a strong causal relationship between NE and GC; an edge cannot exist without its constituent nodes. Standard multitask models often fail to explicitly model this dependency.
  2. Sequence Bias: Knowledge graphs are sets of edges, not ordered sequences. Forcing a model to generate edges in a specific order introduces unnecessary ambiguity and "exposure bias," where early errors propagate through the sequence.

Crake addresses these issues by replacing the sequential decoder with a table-filling mechanism and a causal label-transfer module.

2. Methodology

2.1 Causal-Enhanced Table-Filling (GSG)

Instead of predicting edges one by one, Crake treats the question as an $n \times n$ grid, where $n$ is the number of tokens. It employs Biaffine Attention to predict the probability of an edge existing between any two tokens simultaneously. This approach treats the graph as a set, eliminating the order-dependency of traditional decoders.

To enforce the relationship between node identification and edge creation, Crake introduces a Label Transfer module. This module utilizes a differentiable Gumbel-softmax to inject predictions from the Node Extraction step directly into the Graph Composition step. This ensures the model learns that edges should primarily originate from or terminate at tokens identified as valid entities or variables.

2.2 Beam-Search Relation Extraction (RE)

Identifying the correct predicates in a KB with thousands of properties is computationally expensive. Crake utilizes a BeamSearchRE algorithm. Rather than evaluating all possible graph combinations, it retrieves and ranks candidate predicates for each edge incrementally. This search strategy ensures that the computational complexity scales linearly with the number of edges, making it highly efficient for multi-hop queries.

3. Experimental Setup

We evaluated Crake on LC-QuAD 1.0, a standard open-domain KBQA benchmark based on DBpedia. The dataset consists of 5,000 questions (4,800 training/dev and 1,000 test). To support the table-filling approach, we manually annotated the dataset with node-to-mention mappings, ensuring the model could learn which specific words in a question correspond to nodes in the resulting SPARQL query.

4. Results and Discussion

4.1 Performance

Crake established a new state-of-the-art on LC-QuAD 1.0 with an F1-score of 0.715. This represents a 17% absolute improvement over previous models (0.536) that do not rely on external linguistic tools like dependency parsers.

4.2 Efficiency

The table-filling architecture is remarkably lightweight, requiring only 1/100th of the parameters of traditional sequence-to-sequence decoders. Furthermore, the BeamSearchRE algorithm provided a massive speedup in complex scenarios; for 3-hop queries, Crake processed results in 2.72 seconds, compared to 84.2 seconds for baseline methods—a 30x improvement.

5. Conclusion

Crake demonstrates that treating KBQA as a causal table-filling task is both more accurate and more efficient than traditional sequence generation. By explicitly modeling the dependency between node extraction and graph composition, and utilizing a pruned search for relations, the model handles the scale of DBpedia with high precision. Future work will focus on expanding the model's support for complex SPARQL operators like FILTER and UNION, and reducing the reliance on manual node-mention annotations through semi-supervised learning.

Summary updated

Abstract

Semantic parsing solves knowledge base (KB) question answering (KBQA) by composing a KB query, which generally involves node extraction (NE) and graph composition (GC) to detect and connect related nodes in a query. Despite the strong causal effects between NE and GC, previous works fail to directly model such causalities in their pipeline, hindering the learning of subtask correlations. Also, the sequence-generation process for GC in previous works induces ambiguity and exposure bias, which further harms accuracy. In this work, we formalize semantic parsing into two stages. In the first stage (graph structure generation), we propose a causal-enhanced table-filler to overcome the issues in sequence-modelling and to learn the internal causalities. In the second stage (relation extraction), an efficient beam-search algorithm is presented to scale complex queries on large-scale KBs. Experiments on LC-QuAD 1.0 indicate that our method surpasses previous state-of-the-arts by a large margin (17%) while remaining time and space efficiency. The code and models are available at https://github.com/AOZMH/Crake.
BibTeX
@inproceedings{zhang2022crake,
  author = {Minhao Zhang and Ruoyu Zhang and Yanzeng Li and Lei Zou},
  title = {{Crake: Causal-Enhanced Table-Filler for Question Answering over Large Scale Knowledge Base}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.136},
}
Figure