A Context-aware Attention Network for Interactive Qestion Answering

Huayu Li, Martin Renqiang Min, Yong Ge, Asim KadavOpenReview 2023 · arXiv · OpenReview

TL;DR

This paper introduces a Context-aware Attention Network (CAN) for question answering (QA) that can handle incomplete or ambiguous information by interactively asking clarifying questions. CAN achieves state-of-the-art results on both standard QA and its newly proposed Interactive QA (IQA) datasets by employing a novel two-level attention mechanism and an interactive feedback loop.

What Problem Does This Paper Address?

Traditional question answering (QA) systems, often built on sequence-to-sequence neural networks, excel at extracting answers from well-defined text. However, they struggle in real-world scenarios where information might be incomplete or ambiguous, leading to situations where the system doesn't have enough context to provide a definitive answer. This is particularly common in interactive QA (IQA), where a system needs to communicate with a user to resolve uncertainties. Existing models often fail to recognize these ambiguous states or lack a mechanism to effectively solicit and utilize user feedback to clarify the situation.

Methods

The core of the proposed solution is the Context-aware Attention Network (CAN), an encoder-decoder model enhanced with several key innovations:

  • Two-Level Attention: CAN employs a hierarchical attention mechanism.
  • Word-Level Attention: This attention is context-dependent, meaning the importance of a word is influenced by its surrounding words within a sentence and the overall story context. This helps in capturing nuanced word meanings.
  • Sentence-Level Attention: This attention is guided by the question itself, allowing the model to focus on sentences that are most relevant to answering the query.
  • Context Encoder: This module aggregates information from multiple sentences, considering their sequential order, to build a comprehensive understanding of the story.
  • Interactive Mechanism: This is the most novel aspect. When CAN detects ambiguity or insufficient information to answer a question, it generates a supplementary question to the user. The user's feedback is then encoded and used to update the sentence-level attention weights, allowing the model to refine its understanding and predict the final answer. This feedback loop is crucial for handling IQA tasks.
  • Self-Adaptive Nature: The model can decide whether to directly output an answer or to initiate an interactive clarification, making it "self-adaptive."

Datasets

The authors introduce a new benchmark called ibAbI, designed to specifically test Interactive Question Answering (IQA) capabilities. ibAbI is derived from the well-known Facebook bAbI dataset, which consists of 20 different reasoning tasks. The ibAbI dataset simulates IQA scenarios by creating ambiguous or incomplete versions of bAbI tasks. They generated three specific ibAbI datasets focusing on ambiguous actor, ambiguous object, and information-seeking problems. To evaluate performance in mixed scenarios, they also created sub-datasets by blending IQA data with standard QA data at various ratios (RIQA).

Key Insights and Ideas

  • The Importance of "Knowing What You Don't Know": The paper highlights that a truly intelligent QA system must not only find answers but also recognize when it cannot find an answer due to missing information. This self-awareness is key to effective interaction.
  • Two-Level Attention for Deeper Understanding: The combination of context-aware word-level attention and question-guided sentence-level attention allows CAN to build richer representations of both sentences and the overall story context, leading to more accurate predictions.
  • Interactive Feedback as a Powerful Tool: The interactive mechanism, where the model asks clarifying questions and uses feedback to update its attention, significantly boosts performance on IQA tasks. This is a more robust approach than simply guessing or failing when faced with ambiguity.
  • Bridging the Gap to Real-World QA: By addressing incomplete information, CAN moves closer to building QA systems that can engage in natural, human-like conversations.

Key Results

  • State-of-the-Art Performance: CAN significantly outperforms existing state-of-the-art QA models on both standard QA and the newly introduced IQA datasets.
  • IQA Improvement: On IQA datasets, CAN achieves up to a 40% improvement over conventional QA models that lack an interactive procedure.
  • Robustness to Ambiguity: CAN demonstrates a remarkable ability to handle ambiguous questions, achieving near 0% test error rate on IQA tasks with high IQA ratios.
  • Effective Attention: Qualitative analysis shows that CAN's attention mechanism effectively identifies relevant sentences, especially in long stories, and that user feedback further refines this attention to focus on the most crucial information.
  • Efficient Learning: The interactive mechanism allows the model to adapt to user feedback without requiring re-training of the encoder, making it efficient.

Limitations and Future Work

  • Single Supplementary Question: The current model is limited to generating at most one supplementary question. The authors suggest extending this to multi-round feedback for more complex scenarios.
  • Harder Reasoning Tasks: Future work aims to extend the framework to more challenging co-reference resolution tasks (like the Winograd Schema Challenge) and interactive visual QA tasks that involve greater uncertainty.
  • More Powerful Attention: The authors plan to explore more advanced attention mechanisms and multi-round feedback for even greater self-awareness and adaptability.

Summary updated

Abstract

Neural network based sequence-to-sequence models in an encoder-to-decoder framework have been successfully applied to solve Question Answering (QA) problems, predicting answers from statements and questions. However, almost all previous models have failed to consider detailed context information and unknown states under which systems do not have enough information to answer given questions. These scenarios with incomplete or ambiguous information are very common in the setting of Interactive Question Answering (IQA). To address this challenge, we develop a novel model, employing context-dependent word-level attention for more accurate statement representations and question-guided sentence level attention for better context modeling. We also generate unique IQA datasets to test our model, which will be made publicly available. Employing these attention mechanisms, our model accurately understands when it can output an answer or when it requires generating a supplementary question for additional input depending on different contexts. When available, user’s feedback is encoded and directly applied to update sentence-level attention to infer an answer. Extensive experiments on QA and IQA datasets quantitatively demonstrate the effectiveness of our model with significant improvement over state-of-the-art conventional QA models.
BibTeX
@misc{li2023context,
  author = {Huayu Li and Martin Renqiang Min and Yong Ge and Asim Kadav},
  title = {{A Context-aware Attention Network for Interactive Qestion Answering}},
  year = {2023},
  eprint = {1612.07411},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1612.07411},
  howpublished = {OpenReview},
}
Figure