Knowledge-Enhanced Iterative Instruction Generation and Reasoning for Knowledge Base Question Answering

Haowei Du, Quzhe Huang, Chen Zhang, Dongyan Zhao

Introduction

Multi-hop Knowledge Base Question Answering (KBQA) represents a significant challenge in natural language processing, requiring models to navigate complex graphs of entities and relations to retrieve answers several steps away from a starting point. While retrieval-based methods have shown promise, they often struggle with the complexity of large-scale knowledge bases. This paper introduces KBIGER (Knowledge Base Iterative Instruction GEnerating and Reasoning), a framework designed to mitigate the common pitfalls of multi-hop reasoning by dynamically generating instructions based on the current state of the knowledge graph.

The Challenge: Error Accumulation in Multi-hop Reasoning

The primary obstacle in multi-hop KBQA is error accumulation. Most existing methods rely on a static planning approach: they analyze the natural language question and generate a fixed sequence of reasoning instructions before interacting with the knowledge graph.

Consider the question: "Who does Brian Dawkins play for 2011?" A model must identify the entity "Brian Dawkins," navigate to his roster for 2011, and finally identify the associated team. If a static model incorrectly follows a relation to "Extra salary" instead of "Roster" at the first hop, it has no mechanism to recognize this mistake. Because the subsequent instructions are already fixed, the model cannot backtrack or adjust its path, inevitably leading to an incorrect final answer. This lack of feedback between the reasoning state and the instruction generator is the core problem KBIGER aims to solve.

The KBIGER Framework

KBIGER addresses error accumulation by treating instruction generation and graph reasoning as an interleaved, iterative process. Instead of a one-shot plan, the model generates instructions dynamically at each step.

1. Iterative Instruction Generation

At each step $k$, the model generates a new instruction by considering three inputs: the original natural language question, the previous instruction, and the current state of the reasoning graph. This allows the model to "re-read" the question in the context of where it currently stands in the knowledge base.

2. Knowledge-Enhanced Graph Aggregation

To inform the instruction generator, KBIGER uses an attention-based aggregation mechanism. This component summarizes the current distribution of entities in the knowledge graph into a single vector. This vector acts as a "location signal," telling the generator which parts of the graph are currently being explored and whether the current path aligns with the semantic requirements of the question.

3. Reasoning and Traversal

Using the dynamic instruction, the reasoning component performs a hop across the graph. It employs a Graph Neural Network (GNN) approach to calculate the probability of moving to neighboring entities, selecting paths that best match the current instruction's semantic intent.

The KBIGER framework alternates between generating an instruction and reasoning over the graph, using the N-th graph state to inform the next step.

4. Teacher-Student Framework

Since intermediate reasoning steps are rarely labeled in standard datasets, the authors implement a teacher-student training setup. A "teacher" model, which has access to the ground-truth answer, provides a supervision signal for the intermediate entity distributions, helping the "student" model learn effective navigation strategies.

Experimental Evaluation

The model was evaluated on two prominent multi-hop KBQA benchmarks:

  • WebQuestionsSP (WebQSP): A dataset featuring questions 1-2 hops away from the topic entity.
  • Complex WebQuestions (CWQ): A more challenging benchmark requiring up to 4 hops and involving complex constraints.

KBIGER achieved new state-of-the-art (SOTA) performance on both benchmarks, reaching 75.3% Hits@1 on WebQSP and 50.2% Hits@1 on CWQ. These results represent improvements of 1.0 and 1.4 percentage points, respectively, over previous leading models like NSM.

Analysis: The Power of Revision

The defining characteristic of KBIGER is its ability to perform error revision. By feeding the graph state back into the instruction generator, the model uses the knowledge graph as a compass. If the model reaches an entity that lacks logical outgoing relations to satisfy the remainder of the question, the next generated instruction can pivot to correct the path.

An example of error revision: the model initially follows an incorrect path but uses the revision step to return to the correct 'Roster' entity.

In an analysis of successful cases, the authors found that in 58% of instances where KBIGER succeeded and static baselines failed, it was specifically due to this revision capability. As shown in the Brian Dawkins example, even when the model initially follows a misleading relation (like "Extra salary"), the subsequent instruction step recognizes the dead end and redirects the search back to the "Roster" entity.

Conclusion

KBIGER demonstrates that multi-hop reasoning is most effective when it is a dialogue between language understanding and graph exploration. By moving from static to dynamic planning, the model significantly reduces the impact of error propagation. Future work may involve integrating pre-trained knowledge graph embeddings to further refine the fusion of textual and structural data, as well as optimizing the computational efficiency of the iterative generation process.

Summary updated

BibTeX
@inproceedings{du2022knowledge,
  author = {Haowei Du and Quzhe Huang and Chen Zhang and Dongyan Zhao},
  title = {{Knowledge-Enhanced Iterative Instruction Generation and Reasoning for Knowledge Base Question Answering}},
  booktitle = {NLPCC},
  year = {2022},
  doi = {10.1007/978-3-031-17120-8\_34},
}

Loading related content...

Figure