Unanswerable Question Correction in Question Answering over Personal Knowledge Base

An-Zi Yen, Hen-Hsen Huang, Hsin-Hsi ChenAAAI 2021 · DOI

Introduction

When recalling personal experiences, human memory is often fallible. We frequently conflate details—mixing up which friend accompanied us to an event or which specific location we visited. In the context of Personal Knowledge Bases (PKBs), which serve as digital "lifelogs" of a person's experiences, these inaccuracies lead to "unanswerable" questions. Traditional Knowledge Base Question Answering (KBQA) systems typically fail when faced with such factual inconsistencies, either returning no result or failing silently.

This research introduces the task of Personal Knowledge Base Question Answering with Correction (PKBQAC). Instead of simply rejecting an erroneous query, the proposed system identifies the factual error and suggests a corrected, answerable version based on the user's actual history stored in the PKB.

Example of a user asking an unanswerable question and the system providing corrected suggestions based on a Knowledge Graph.

The PKBQAC Framework

The authors propose a two-stage framework to handle unanswerable questions through recognition and subsequent correction.

The two-stage workflow: Stage 1 focuses on unanswerable recognition and Stage 2 handles question generation for suggestions.

Stage 1: Unanswerable Recognition (GWSMNet)

The first stage involves determining whether a question can be answered by the PKB. The system utilizes a Graph Weighted Semantic Matching Network (GWSMNet). Unlike standard text-based models, GWSMNet incorporates the structural relationships of the Knowledge Base using GraphSAGE. By calculating a similarity score between the question and candidate "query graphs" (subsections of the KB), the model can distinguish between a "hard" answerable question and one that is fundamentally unanswerable due to factual contradiction. If the similarity score falls below a threshold (0.5), the question is flagged for correction.

Stage 2: Question Generation (GenTagNet)

Once a question is flagged as unanswerable, the system identifies the most likely intended facts in the KB and generates a corrected question. The authors introduce GenTagNet, which approaches correction as a sequence tagging task (Keep, Delete, or Add) rather than full-sentence generation. This "editing" approach preserves the user's original phrasing and intent. GenTagNet is trained using Reinforcement Learning (RL), with a reward function designed to balance linguistic fluency (BLEU score) with "answerability"—ensuring the suggested question is factually grounded in the KB.

Dataset and Evaluation

To support this research, the authors developed the PKBQAC dataset, derived from the LiveKB corpus. It contains approximately 42,000 questions (including 3,759 unanswerable ones) linked to 15,500 real-life events. The dataset covers varying levels of complexity, from simple single-relation questions to complex queries linking multiple events.

Key Results

  • Recognition: GWSMNet achieved an MRR of 0.8212 on answerable questions and 69.89% accuracy in identifying unanswerable ones, outperforming BERT-based baselines.
  • Generation: The RL-based GenTagNet achieved a BLEU4 score of 45.18, significantly higher than standard Seq2Seq Transformers (40.01).
  • Human Preference: In blind evaluations, human annotators preferred GenTagNet's suggestions 43.5% of the time, noting higher fluency and reasonableness compared to pipeline baselines.

Conclusion and Future Directions

The core insight of this work is that structural KB information is vital for recognizing user errors, and editing an existing query is more effective than generating a new one from scratch. While highly effective for personal lifelogs, future work aims to extend this to multi-turn dialogues and address challenges related to KB incompleteness, where the system might incorrectly "correct" a user when the database itself is missing information.

Summary updated

BibTeX
@inproceedings{yen2021unanswerable,
  author = {An-Zi Yen and Hen-Hsen Huang and Hsin-Hsi Chen},
  title = {{Unanswerable Question Correction in Question Answering over Personal Knowledge Base}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i16.17678},
}
Figure