Auto-ACE: An Automatic Answer Correctness Evaluation Method for Conversational Question Answering

Zhixin Bai, Bingbing Wang, Bin Liang, Ruifeng XuACL 2024

TL;DR

This paper introduces Auto-ACE, a novel framework for conversational question answering (ConvQA) that tackles the critical mismatch between training with perfect answers and real-world inference with imperfect, predicted answers. By employing an Att-BERT model to weigh historical Q&A pairs and an A-Scorer to evaluate predicted answer confidence, Auto-ACE significantly improves performance and robustness.

What Problem Does This Paper Address?

Conversational Question Answering (ConvQA) systems aim to answer questions within a dialogue, leveraging both the current context and the history of previous questions and answers. A major challenge in developing these systems is the discrepancy between how they are trained and how they operate in the real world. During training, models often use "ground-truth" answers from the conversation history. However, in actual use, these historical answers are not always correct; they are predictions from previous steps in the conversation. Simply feeding these potentially incorrect predicted answers into the model during training can introduce "noise," degrading performance. Existing methods that try to address this either still rely on ground truth or require complex multi-stage training, increasing computational cost. This paper aims to bridge this gap by creating a method that effectively uses predicted answers during training without sacrificing performance.

Methods

The proposed Auto-ACE framework has two core components:

  • Att-BERT: This model enhances BERT by incorporating an attention mechanism. It learns to assign attention weights to historical question-answer pairs based on their relevance to the current question. This helps the model focus on the most pertinent parts of the conversation history. The attention weight for a question is determined by its similarity to the current question.
  • A-Scorer: This is a novel answer scorer designed to evaluate the confidence of predicted answers. It takes the current question, the predicted answer, and the context as input. The A-Scorer's output is then used to further refine the attention weights assigned by Att-BERT. Specifically, it down-weights or completely ignores predicted answers that are deemed low-confidence or incorrect. This is crucial for minimizing the impact of noisy predictions.

These two components are trained jointly, allowing them to work together seamlessly during both training and inference.

Datasets

The paper evaluates Auto-ACE on two widely used ConvQA benchmarks:

  • QuAC (Question Answering in Context): This dataset simulates realistic information-seeking conversations.
  • CoQA (Conversational Question Answering): Another benchmark dataset for conversational question answering.

The authors use the development sets of both QuAC and CoQA for evaluation, as their test sets are not publicly available.

Key Insights and Ideas

  • The "Ground Truth vs. Prediction" Dilemma: The paper highlights a fundamental problem in ConvQA: training models with perfect historical answers leads to a performance gap when deployed in real-world scenarios where historical answers are predictions and can be wrong.
  • Confidence is Key: Not all predicted answers are equally reliable. The A-Scorer's ability to gauge the confidence of these predictions is crucial for filtering out noise and focusing on useful information.
  • Attention as a Noise Filter: The attention mechanism, particularly when guided by confidence scores, acts as a powerful tool to selectively ignore or down-weight irrelevant or incorrect historical information, preventing it from negatively impacting the current prediction.
  • Single-Stage Training Advantage: Auto-ACE's ability to train Att-BERT and A-Scorer jointly in a single stage is a significant advantage over methods that require multiple training and prediction steps.

Key Results

  • Auto-ACE demonstrates significant performance improvements over various baselines on both QuAC and CoQA datasets, achieving the best results among methods that do not rely on ground-truth answers.
  • The proposed method outperforms the strongest baseline that uses predicted answers by 1.32% on the F1-score metric.
  • Ablation studies confirm the necessity of both the attention mechanism (for questions and answers) and the A-Scorer for achieving optimal performance.
  • The paper also shows that while longer conversation histories can be beneficial, the most recent turns (1-3) tend to be the most relevant, and Auto-ACE's attention mechanism helps it manage longer histories more effectively than models without attention.

Limitations and Future Work

  • Handling Long Conversations: The authors acknowledge that the model's ability to process very long conversational histories needs further enhancement to maintain consistent performance. Future work could explore richer representations of real-world conversations.
  • A-Scorer Robustness: While effective, the A-Scorer might still introduce noise if it misjudges the confidence of predicted answers. The authors suggest that employing large language models (LLMs) could further improve the accuracy of answer evaluation.

Summary updated

BibTeX
@inproceedings{bai2024auto,
  author = {Zhixin Bai and Bingbing Wang and Bin Liang and Ruifeng Xu},
  title = {{Auto-ACE: An Automatic Answer Correctness Evaluation Method for Conversational Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure