Improving Complex Knowledge Base Question Answering via Question-to-Action and Question-to-Question Alignment

Yechun Tang, Xiaoxia Cheng, Weiming Lu

Abstract

Knowledge Base Question Answering (KBQA) for complex queries remains a significant challenge due to the semantic and structural gap between natural language and logical action sequences. We introduce ALCQA (Alignment-enhanced Complex Question Answering), a framework that bridges this gap through two primary alignment strategies: question-to-action alignment via structured rewriting and question-to-question alignment via similarity-based validation. Evaluated on the CQA dataset, ALCQA achieves an 80.89% Macro F1 score, a 9.88% improvement over previous state-of-the-art models.

Introduction

The goal of KBQA is to translate natural language questions into executable logical queries. While simple one-hop questions are largely solved, complex questions involving logical reasoning, quantitative counting, and comparative constraints present major hurdles. The core difficulty lies in the fluidity of human language compared to the rigid, hierarchical nature of database action sequences. Existing models often struggle to map these disparate representations, particularly when high-quality logical form annotations are scarce.

Methodology

The ALCQA framework decomposes the KBQA task into three distinct stages to ensure robust alignment between the question and the resulting action sequence.

The overall architecture of the ALCQA framework, illustrating the flow from question encoding to action sequence selection.

1. Question Rewriting (Question-to-Action Alignment)

To bridge the structural gap, ALCQA first "rewrites" complex natural language questions into a series of simpler, structured utterances separated by a # symbol. Each utterance corresponds to a single database action. Because datasets for such rewritten questions are typically unavailable, the authors employ a two-phase back-translation technique. A model is trained to translate database actions back into natural language, which is then used to generate synthetic training data for the rewriter.

The question rewriting process using back-translation to generate structured utterances from action sequences.

2. Candidate Generation

A sequence-to-sequence (Seq2Seq) model, utilizing a BERT and BiLSTM encoder paired with an LSTM decoder, generates potential action sequences via beam search. A key feature of this stage is the use of BERT to perform cross-attention between the input question and Knowledge Graph (KG) artifacts, such as entities and relations, facilitating the selection of correct database terms.

3. Action Sequence Selection (Question-to-Question Alignment)

In the final phase, the model performs a "sanity check" by leveraging a memory of similar question-answer pairs (a support set) from the training data. Candidate action sequences are validated by substituting entities from similar questions and executing them. If a candidate sequence yields the correct answer for a similar question, its score is increased, reinforcing the logical consistency of the selection.

Experimental Results

ALCQA was evaluated on two benchmarks: the Complex Sequential Question Answering (CQA) dataset and WebQuestionsSP (WQSP).

  • CQA Performance: ALCQA achieved an overall Macro F1 of 80.89%, outperforming the previous best model, NS-CQA, by 9.88%. The most significant gains were observed in difficult categories such as "Comparative Reasoning" (improving from 65.87% to 83.09%) and "Comparative (Count)" (improving from 43.25% to 60.80%).
  • WQSP Performance: On simpler tasks, the model maintained a competitive F1 score of 73.6%, demonstrating its versatility across varying levels of question complexity.

Conclusion

The ALCQA framework demonstrates that decomposing complex questions into structured sub-tasks and validating logic through similarity-based alignment significantly improves KBQA performance. While the model shows state-of-the-art results, it remains dependent on the accuracy of initial entity linking. Future work may explore more sophisticated semantic similarity models for support set selection and hybrid approaches combining sequence-based logic with graph-based pathfinding.

Summary updated

Abstract

Complex knowledge base question answering can be achieved by converting questions into sequences of predefined actions. However, there is a significant semantic and structural gap between natural language and action sequences, which makes this conversion difficult. In this paper, we introduce an alignment-enhanced complex question answering framework, called ALCQA, which mitigates this gap through question-to-action alignment and question-to-question alignment. We train a question rewriting model to align the question and each action, and utilize a pretrained language model to implicitly align the question and KG artifacts. Moreover, considering that similar questions correspond to similar action sequences, we retrieve top-k similar question-answer pairs at the inference stage through question-to-question alignment and propose a novel reward-guided action sequence selection strategy to select from candidate action sequences. We conduct experiments on CQA and WQSP datasets, and the results show that our approach outperforms state-of-the-art methods and obtains a 9.88\% improvements in the F1 metric on CQA dataset. Our source code is available at https://github.com/TTTTTTTTy/ALCQA.
BibTeX
@inproceedings{tang2022improving,
  author = {Yechun Tang and Xiaoxia Cheng and Weiming Lu},
  title = {{Improving Complex Knowledge Base Question Answering via Question-to-Action and Question-to-Question Alignment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=dF78QjLq9H0},
}
Figure