Abstract
Factoid question-answering (QA) systems often struggle with "silent failures," where they provide incorrect answers without any internal mechanism for verification. This paper presents QUESREV (Question Revision), a post-inspection framework designed to recover from such errors. QUESREV transforms the original question into a set of "revised queries" by integrating information from predicted Knowledge Base (KB) relations. By evaluating the semantic coherence of these revised queries using a deep learning model, the system can effectively rerank candidate answers. Experimental results on the WebQuestions benchmark demonstrate that QUESREV consistently improves the performance of existing QA systems, achieving state-of-the-art results when combined with ensemble methods.
Introduction
Answering factual questions using large-scale knowledge bases (KBs) like Freebase is a fundamental task in natural language processing. While modern semantic parsers have made significant strides, they frequently fail to detect when a predicted relation is logically inconsistent with the user's intent. For example, if a system is asked "What did Mary Wollstonecraft fight for?" and incorrectly predicts her profession (writer) instead of her area of activism (women's rights), it lacks a "sanity check" to realize that "What profession did Mary Wollstonecraft fight for?" is a semantically awkward interpretation.
This paper addresses this gap by introducing a post-inspection component that revises the original question based on predicted KB relations. This approach allows the system to verify if the resulting "revised query" makes sense in natural language, thereby identifying and correcting errors made during the initial parsing phase.
Methods: The QUESREV Framework
The authors propose QUESREV, a "plug-and-play" component that can be integrated with existing QA systems. The framework operates by generating and scoring revised versions of the input question.
Query Revision Strategies
The core of QUESREV lies in its ability to transform a question into a template and populate it with evidence from the KB. The authors define three distinct levels of revision, varying in their degree of abstraction and expressivity:
- Entity-Centric (EC): This strategy replaces the specific entity in the question with its general type (e.g., "What did [activist] fight for?").
- Answer-Centric (AC): This level adds the expected answer type to the "wh-word" to clarify the target of the question (e.g., "[What activism issue] did [activist] fight for?").
- Relation-Centric (RC): The most expressive strategy, it plugs the actual KB relation description directly into the question structure (e.g., "[What area of activism] did [activist] fight for?").
Model Architecture and Integration
QUESREV utilizes a Bidirectional LSTM (Bi-LSTM) to encode these revised questions into vector representations. A scoring function then determines the "naturalness" or consistency of each revision. The model is trained using a hinge-loss objective, learning to assign higher scores to revisions derived from correct KB relations and lower scores to incorrect ones.
In practice, QUESREV acts as a reranker. It evaluates the top candidates produced by a base QA system (such as STAGG) and can promote a lower-ranked candidate if its corresponding revised query exhibits higher semantic coherence.
Experimental Setup
The framework was evaluated primarily on the WEBQUESTIONS benchmark, which contains 5,810 questions mapped to Freebase. To combat data sparsity and help the model learn the relationship between natural language and KB schemas, the authors utilized SIMPLEQUESTIONS (a dataset of 108k pairs) for data augmentation.
Results and Discussion
The addition of QUESREV provided a significant performance boost across various configurations. When applied to the STAGG system, it improved the $F_1$ score from 52.5% to 53.9%.
Comparison with State-of-the-Art
As shown in the table below, QUESREV on STAGG outperforms several complex end-to-end models. When combined with an ensemble approach (STAGG-RANK), the system achieved a state-of-the-art $F_1$ score of 54.3%.
Key Insights
- Structural Blending: The authors found that "blending" KB evidence directly into the question structure is more effective than parallel encoding (where the question and evidence are processed separately). This preservation of grammatical flow makes it easier for the Bi-LSTM to detect inconsistencies.
- Granularity Trade-offs: While Relation-Centric (RC) revisions are the most powerful, they require substantial training data to generalize. In contrast, Answer-Centric (AC) revisions are more robust when data is scarce.
Conclusion
QUESREV demonstrates that post-inspection via query revision is a powerful technique for recovering from errors in factoid QA. By transforming the task into a verification problem, the system can leverage linguistic patterns to ensure logical consistency. While the system's effectiveness is currently limited by the quality of the initial candidate list, future work could extend this logic to more complex, multi-hop reasoning tasks.
Summary updated