Recovering Question Answering Errors via Query Revision

Semih Yavuz, Izzeddin Gur, Yu Su, Xifeng YanEMNLP 2017 · OpenReview

Abstract

Factoid question-answering (QA) systems often struggle with "silent failures," where they provide incorrect answers without any internal mechanism for verification. This paper presents QUESREV (Question Revision), a post-inspection framework designed to recover from such errors. QUESREV transforms the original question into a set of "revised queries" by integrating information from predicted Knowledge Base (KB) relations. By evaluating the semantic coherence of these revised queries using a deep learning model, the system can effectively rerank candidate answers. Experimental results on the WebQuestions benchmark demonstrate that QUESREV consistently improves the performance of existing QA systems, achieving state-of-the-art results when combined with ensemble methods.

Introduction

Answering factual questions using large-scale knowledge bases (KBs) like Freebase is a fundamental task in natural language processing. While modern semantic parsers have made significant strides, they frequently fail to detect when a predicted relation is logically inconsistent with the user's intent. For example, if a system is asked "What did Mary Wollstonecraft fight for?" and incorrectly predicts her profession (writer) instead of her area of activism (women's rights), it lacks a "sanity check" to realize that "What profession did Mary Wollstonecraft fight for?" is a semantically awkward interpretation.

This paper addresses this gap by introducing a post-inspection component that revises the original question based on predicted KB relations. This approach allows the system to verify if the resulting "revised query" makes sense in natural language, thereby identifying and correcting errors made during the initial parsing phase.

Methods: The QUESREV Framework

The authors propose QUESREV, a "plug-and-play" component that can be integrated with existing QA systems. The framework operates by generating and scoring revised versions of the input question.

Query Revision Strategies

The core of QUESREV lies in its ability to transform a question into a template and populate it with evidence from the KB. The authors define three distinct levels of revision, varying in their degree of abstraction and expressivity:

  1. Entity-Centric (EC): This strategy replaces the specific entity in the question with its general type (e.g., "What did [activist] fight for?").
  2. Answer-Centric (AC): This level adds the expected answer type to the "wh-word" to clarify the target of the question (e.g., "[What activism issue] did [activist] fight for?").
  3. Relation-Centric (RC): The most expressive strategy, it plugs the actual KB relation description directly into the question structure (e.g., "[What area of activism] did [activist] fight for?").
Three levels of query revision: Entity-Centric (EC), Answer-Centric (AC), and Relation-Centric (RC).

Model Architecture and Integration

QUESREV utilizes a Bidirectional LSTM (Bi-LSTM) to encode these revised questions into vector representations. A scoring function then determines the "naturalness" or consistency of each revision. The model is trained using a hinge-loss objective, learning to assign higher scores to revisions derived from correct KB relations and lower scores to incorrect ones.

In practice, QUESREV acts as a reranker. It evaluates the top candidates produced by a base QA system (such as STAGG) and can promote a lower-ranked candidate if its corresponding revised query exhibits higher semantic coherence.

Experimental Setup

The framework was evaluated primarily on the WEBQUESTIONS benchmark, which contains 5,810 questions mapped to Freebase. To combat data sparsity and help the model learn the relationship between natural language and KB schemas, the authors utilized SIMPLEQUESTIONS (a dataset of 108k pairs) for data augmentation.

Results and Discussion

The addition of QUESREV provided a significant performance boost across various configurations. When applied to the STAGG system, it improved the $F_1$ score from 52.5% to 53.9%.

Comparison with State-of-the-Art

As shown in the table below, QUESREV on STAGG outperforms several complex end-to-end models. When combined with an ensemble approach (STAGG-RANK), the system achieved a state-of-the-art $F_1$ score of 54.3%.

Comparison of QUESREV with state-of-the-art question answering systems on the WebQuestions dataset.

Key Insights

  • Structural Blending: The authors found that "blending" KB evidence directly into the question structure is more effective than parallel encoding (where the question and evidence are processed separately). This preservation of grammatical flow makes it easier for the Bi-LSTM to detect inconsistencies.
  • Granularity Trade-offs: While Relation-Centric (RC) revisions are the most powerful, they require substantial training data to generalize. In contrast, Answer-Centric (AC) revisions are more robust when data is scarce.

Conclusion

QUESREV demonstrates that post-inspection via query revision is a powerful technique for recovering from errors in factoid QA. By transforming the task into a verification problem, the system can leverage linguistic patterns to ensure logical consistency. While the system's effectiveness is currently limited by the quality of the initial candidate list, future work could extend this logic to more complex, multi-hop reasoning tasks.

Summary updated

Abstract

In the context of embodied artificial intelligence, morphological computation refers to processes which are conducted by the body (and environment) that otherwise would have to be performed by the brain. Exploiting environmental and morphological properties is an important feature of embodied systems. The main reason is that it allows to significantly reduce the controller complexity. An important aspect of morphological computation is that it cannot be assigned to an embodied system per se, but that it is, as we show, behavior- and state-dependent. In this work, we evaluate two different measures of morphological computation that can be applied in robotic systems and in computer simulations of biological movement. As an example, these measures were evaluated on muscle and DC-motor driven hopping models. We show that a state-dependent analysis of the hopping behaviors provides additional insights that cannot be gained from the averaged measures alone. This work includes algorithms and computer code for the measures.
BibTeX
@inproceedings{yavuz2017recovering,
  author = {Semih Yavuz and Izzeddin Gur and Yu Su and Xifeng Yan},
  title = {{Recovering Question Answering Errors via Query Revision}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=BJVb5GM_bB},
}
Figure