Answer Credibility: A Language Modeling Approach to Answer Validation

Protima Banerjee, Hyoil HanNAACL 2009

TL;DR

This paper introduces "Answer Credibility," a novel language modeling approach to assess the reliability of a source document containing a candidate answer in Question Answering (QA) systems. By quantifying the semantic similarity between a question's context and the source document, this method aims to improve answer validation without relying on external knowledge sources.

What Problem Does This Paper Address?

The field of Question Answering (QA) systems aims to provide direct answers to user queries. A critical component of robust QA systems is "Answer Validation," which determines if a proposed answer is indeed correct and supported by the retrieved text. Existing methods often rely on external web sources or complex textual entailment techniques, which can be computationally expensive or not applicable to all question types. This paper addresses the need for an efficient, self-contained method to validate answers by focusing on the inherent credibility of the source document itself, relative to the question's context.

Methods

The core contribution is the development of an "Answer Credibility" score. This score is a similarity measure between two language models:

  • Question Context Model: This model represents the semantic meaning of the question. It's built using an "Aspect-Based Relevance Language Model" (ABRLM), which leverages Probabilistic Latent Semantic Analysis (PLSA) to identify latent aspects of the information need and then constructs a distribution of terms based on their relevance to these aspects.
  • Source Document Model: This is a standard language model derived from the document that contains the candidate answer.

The similarity between these two models is computed using the Kullback-Leibler (KL) divergence. This Answer Credibility score is then used to modulate the original answer score from the QA system through an interpolation technique, effectively boosting the score of answers from more credible sources.

Datasets

The experiments were conducted using the set of all factoid questions from the Text Retrieval Conference (TREC) 2006 Question Answering Track. The OpenEphyra Question Answering testbed was used as the framework for implementation.

Key Insights and Ideas

  • Source Credibility Matters: The paper highlights that the reliability of the source document itself is a crucial, often overlooked, factor in answer validation.
  • Semantic Context is Key: By modeling the "semantic Question Context," the approach captures the underlying meaning of the question, allowing for a more nuanced comparison with the source document.
  • Language Modeling Power: The paper demonstrates the effectiveness of corpus-based language modeling techniques for answer validation, avoiding the need for external knowledge bases.
  • Applicability to Diverse Questions: Unlike some textual entailment methods, this approach is designed to be applicable to a wider range of question types, including those where generating a hypothesis is difficult.

Key Results

The proposed Answer Credibility filter was integrated into the OpenEphyra QA system. The results showed performance improvements in both accuracy and Mean Reciprocal Rank (MRR):

  • MRR Improvements: A 9% improvement for "what"-type questions and an overall 25% improvement for "who"-type questions (including "who," "who is," and "who was"). Overall, 58% of categories performed at or better than the baseline.
  • Accuracy Improvements: A 5% improvement in accuracy for "what"-type questions and a 13% improvement for "who"-type questions.
  • Negative Impact: For question types like "how much" and "how many," where correct answers or documents were often missing from the initial retrieval list, the Answer Credibility filter had a negative effect, as it couldn't boost the rank of non-existent correct answers.

Limitations and Future Work

  • Limitations: The method showed a negative impact on question types where the correct answer or document was not retrieved by the baseline system (e.g., "how much," "how many"). This suggests the approach is most effective when there is supporting information available in the retrieved documents. It also notes that the applicability to questions requiring a date response is limited.
  • Future Work: The authors plan to further develop the Answer Credibility model to incorporate not only terms from the Question Context but also terms that can be deduced to be in an "Answer Context."

Summary updated

BibTeX
@inproceedings{banerjee2009answer,
  author = {Protima Banerjee and Hyoil Han},
  title = {{Answer Credibility: A Language Modeling Approach to Answer Validation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure