Joint Models for Answer Verification in Question Answering Systems

Zeyu Zhang, Thuy Vu, Alessandro MoschittiACL 2021

TL;DR

This paper introduces novel "joint models" for Answer Sentence Selection (AS2) in question answering systems. By treating other answer candidates not just as independent options but as potential sources of support or refutation for a target answer, their models achieve state-of-the-art results on several benchmarks. This approach moves beyond traditional pointwise scoring to a more holistic understanding of answer correctness.

What Problem Does This Paper Address?

Retrieval-based question answering (QA) systems often rely on an Answer Sentence Selection (AS2) module to pick the best answer from a list of candidates provided by an initial retrieval step. Traditional AS2 models typically score each candidate answer independently (pointwise). However, the authors argue that this is suboptimal because other candidate answers can provide valuable context. For instance, one candidate might explicitly support another, or another might contradict it. The challenge is to effectively model these inter-candidate relationships to improve the accuracy of the final answer selection.

Methods

The paper proposes three main approaches:

  • Answer Support-based Reranker (ASR): This is the core novel method. It takes a target answer candidate and considers all other candidates as "evidences." It uses a transformer model to represent the relationship between the target answer and each other candidate. An Answer Support Classifier (ASC) is trained to determine if a candidate supports, refutes, or is neutral towards the target answer. These relationships are then aggregated (e.g., via max-pooling) and combined with the original target answer representation to produce a final score.
  • Joint Model Multi-classifier (MASR): This extends ASR by jointly modeling all top-k candidate answers. It essentially runs multiple ASR models in parallel, one for each potential target answer, and then uses a final classifier to select the best answer from the entire set.
  • KGAT Adaptation: The authors adapt a graph-based fact verification model (KGAT) for AS2. In this approach, candidate answers are treated as "evidences" for a "claim" (the question-answer pair). A graph is constructed where nodes represent these claim-evidence pairs, and edges capture their relationships, allowing for joint reasoning.

Datasets

The models were evaluated on three datasets:

  • WikiQA: A popular dataset for AS2, with questions and candidate answers from Bing query logs over Wikipedia.
  • TREC-QA: Another well-established benchmark for question answering.
  • WQA: A real-world dataset created by Alexa AI, designed to improve understanding and benchmarking of QA systems. It features questions from the web, retrieved documents, and manually assessed candidate answers.

The paper also mentions using the FEVER dataset for pre-training the Answer Support Classifier (ASC) in some MASR variants, leveraging its fact verification labels.

Key Insights and Ideas

  • Inter-candidate relationships matter: The most significant insight is that other answer candidates are not just noise but can provide crucial contextual information (support, refutation, neutrality) for evaluating a target answer.
  • Fact verification techniques are transferable: The success of adapting KGAT, a fact verification model, suggests that the underlying principles of verifying claims using evidence can be applied to verifying answer correctness using other answer candidates.
  • Joint modeling improves performance: Moving from pointwise scoring to joint modeling (ASR and MASR) consistently leads to performance gains, especially on larger datasets like WQA.
  • Pre-training ASC with FEVER data is beneficial: Using FEVER labels to pre-train the Answer Support Classifier (ASC) in MASR-FP leads to improved accuracy, highlighting the value of leveraging related tasks.

Key Results

  • ASR establishes a new state-of-the-art: ASR consistently outperforms previous state-of-the-art models (like TANDA) across all datasets and measures, achieving significant improvements in accuracy (e.g., ~3% absolute P@1 on WikiQA, ~6% on TREC-QA, leading to a 60% error reduction).
  • MASR further improves performance: MASR, by jointly modeling all candidates, provides an additional boost over ASR, particularly on the WQA dataset, demonstrating the benefit of global reasoning.
  • KGAT adaptation shows promise: The adapted KGAT model also improves upon baseline rerankers, confirming the utility of graph-based joint reasoning.
  • Statistical significance: The paper reports statistically significant improvements for ASR over baselines using randomization tests.

Limitations and Future Work

  • Data dependency for MASR: MASR's performance is sensitive to the size of the training corpus. It underperforms on smaller datasets like WikiQA and TREC-QA compared to ASR, suggesting that its larger parameter count requires more data to train effectively.
  • Hyperparameter tuning: The selection of 'k' (the number of candidates to consider for support) is a tricky hyperparameter that risks overfitting on validation sets. The paper proposes a method to mitigate this but acknowledges the challenge.
  • Future work: The authors suggest exploring more sophisticated ways to model inter-candidate relationships and potentially incorporating external knowledge sources to further enhance answer verification. The successful pre-training of ASC with FEVER data also opens avenues for leveraging other related tasks.

Summary updated

BibTeX
@inproceedings{zhang2021joint,
  author = {Zeyu Zhang and Thuy Vu and Alessandro Moschitti},
  title = {{Joint Models for Answer Verification in Question Answering Systems}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure