Improving Complex Knowledge Base Question Answering via Question-to-Action and Question-to-Question Alignment

Yechun Tang, Xiaoxia Cheng, Weiming Lu

Abstract

Complex knowledge base question answering can be achieved by converting questions into sequences of predefined actions. However, there is a significant semantic and structural gap between natural language and action sequences, which makes this conversion difficult. In this paper, we introduce an alignment-enhanced complex question answering framework, called ALCQA, which mitigates this gap through question-to-action alignment and question-to-question alignment. We train a question rewriting model to align the question and each action, and utilize a pretrained language model to implicitly align the question and KG artifacts. Moreover, considering that similar questions correspond to similar action sequences, we retrieve top-k similar question-answer pairs at the inference stage through question-to-question alignment and propose a novel reward-guided action sequence selection strategy to select from candidate action sequences. We conduct experiments on CQA and WQSP datasets, and the results show that our approach outperforms state-of-the-art methods and obtains a 9.88% improvements in the F1 metric on CQA dataset.

Introduction

Knowledge Base Question Answering (KBQA) is the process of teaching machines to answer natural language queries using structured databases, or Knowledge Graphs (KGs). While simple, single-hop questions (e.g., "Who founded Microsoft?") are largely considered solved, Complex CQA remains a significant hurdle. These queries require multiple reasoning steps, such as counting, comparisons, and logical intersections—for example, "Which films had their voice dubbing done by exactly 20 people?"

The core challenge lies in the semantic and structural gap between fluid human language and the rigid, functional "actions" a computer must execute to query a KG. Most existing models struggle to bridge this gap, particularly when they lack explicit "gold" training data that maps every question to a perfect logical form. The ALCQA framework addresses this by introducing two layers of alignment: question-to-action and question-to-question.

The ALCQA Framework

ALCQA (Alignment-enhanced Complex Question Answering) decomposes the problem into three distinct stages designed to simplify the translation from language to logic.

1. Question-to-Action Alignment (Rewriting)

Instead of attempting to parse a complex sentence into a logical program in one pass, ALCQA uses a Question Rewriting model. This model breaks a query into a sequence of simpler utterances, where each segment corresponds to a single logical action. Because direct training data for this mapping is scarce, the authors developed a two-phase training method using back-translation: they converted existing action sequences into questions and compared them to the original text to learn which specific words trigger which actions.

2. Candidate Action Sequence Generation

Using a BERT-based encoder and an LSTM decoder, the system generates multiple potential "action sequences" (logical programs). This stage utilizes Pretrained Language Models (PLMs) to implicitly align question words with specific KG artifacts, such as entities and relations, through a cross-attention mechanism. This ensures the decoder focuses on the correct linguistic cues for each step of the program.

3. Question-to-Question Alignment (Selection)

This stage introduces a novel Reward-guided Action Sequence Selection strategy. During inference, the model retrieves the top-k most similar questions from a pre-built memory of question-answer pairs. It then "tests" its candidate action sequences against these similar historical questions. If a candidate sequence produces a correct answer for a similar question, it receives a higher reward score. This effectively uses the "memory" of the system to validate logical interpretations before finalizing an answer.

Experimental Results

The researchers evaluated ALCQA on two major benchmarks: the CQA (Complex Sequential Question Answering) dataset, which contains over 944,000 training pairs, and the WQSP (WebQuestionsSP) dataset for real-world versatility.

ALCQA significantly outperformed previous state-of-the-art (SOTA) methods, achieving an 80.89% Macro F1 score on the CQA dataset—a 9.88% improvement over the previous best model (NS-CQA). The gains were most pronounced in high-difficulty categories:

  • Quantitative Reasoning: F1 score improved from 56.28% to 76.30%.
  • Comparative (Count): F1 score rose from 43.25% to 60.80%.

Ablation studies confirmed that both alignment strategies are essential. Removing the Question Rewriting module led to a 1.79% performance drop, while removing the Reward-guided Selection module caused a 2.49% drop.

Conclusion and Future Work

ALCQA demonstrates that bridging the semantic gap in KBQA requires both structural decomposition (rewriting) and historical validation (similarity-based selection). By aligning natural language more closely with logical actions, the framework sets a new standard for complex reasoning over knowledge graphs. Future research will focus on improving the efficiency of the preprocessing pipeline and enhancing entity linking to handle even larger and more diverse knowledge bases.

Summary updated

Abstract

Complex knowledge base question answering can be achieved by converting questions into sequences of predefined actions. However, there is a significant semantic and structural gap between natural language and action sequences, which makes this conversion difficult. In this paper, we introduce an alignment-enhanced complex question answering framework, called ALCQA, which mitigates this gap through question-to-action alignment and question-to-question alignment. We train a question rewriting model to align the question and each action, and utilize a pretrained language model to implicitly align the question and KG artifacts. Moreover, considering that similar questions correspond to similar action sequences, we retrieve top-k similar question-answer pairs at the inference stage through question-to-question alignment and propose a novel reward-guided action sequence selection strategy to select from candidate action sequences. We conduct experiments on CQA and WQSP datasets, and the results show that our approach outperforms state-of-the-art methods and obtains a 9.88\% improvements in the F1 metric on CQA dataset. Our source code is available at https://github.com/TTTTTTTTy/ALCQA.
BibTeX
@inproceedings{tang2022improving,
  author = {Yechun Tang and Xiaoxia Cheng and Weiming Lu},
  title = {{Improving Complex Knowledge Base Question Answering via Question-to-Action and Question-to-Question Alignment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=dF78QjLq9H0},
}
Figure