Abstract
In Conversational Question Answering (ConvQA), models are typically trained and evaluated using ground-truth history. However, in realistic deployment scenarios, models must rely on their own previous predictions, which may contain errors. This discrepancy, known as exposure bias, often leads to error propagation where a single incorrect answer degrades the performance of all subsequent turns. We present AS-ConvQA, a framework that employs calibrated confidence and uncertainty measurements to selectively filter conversation history. By using temperature scaling and Monte Carlo (MC) Dropout, the model identifies and excludes untrustworthy predictions. Our results on the QuAC and CoQA benchmarks show that AS-ConvQA effectively mitigates error accumulation, outperforming standard baselines and maintaining robustness throughout extended dialogues.
1. Introduction
Conversational Question Answering (ConvQA) requires a model to understand the context of a multi-turn dialogue to answer sequential questions. While standard benchmarks provide "gold" (ground-truth) answers for previous turns to assist the model, real-world applications do not have this luxury. In practice, a model must use its own previous outputs as context. This creates a significant challenge: if the model provides an incorrect answer in an early turn, that error is treated as fact in subsequent turns, leading to a "snowball effect" of failure. This paper addresses this "exposure bias" by introducing a selection mechanism that filters the model's history based on its own self-assessed reliability.
2. The Challenge of Noisy History
Research indicates a "No History" paradox in ConvQA: models often perform better by ignoring their own previous predictions entirely ("No Pred") rather than using a history filled with unvetted, potentially incorrect answers ("All Pred"). This suggests that noisy history is more detrimental to model reasoning than a lack of history. To bridge the gap between idealized lab settings and realistic deployment, a model must be able to distinguish between its correct and incorrect predictions.
3. AS-ConvQA: Framework and Methodology
The proposed Answer Selection-based realistic ConvQA (AS-ConvQA) framework treats the model's previous predictions as candidates for the conversation history. It employs two primary metrics to decide whether to include a prediction in the context for the next turn:
3.1 Confidence and Uncertainty Measurement
- Confidence: This is calculated as the maximum softmax probability of the predicted answer span.
- Uncertainty: To capture the model's internal doubt, the researchers use Monte Carlo (MC) Dropout. By performing multiple stochastic forward passes with different dropout masks, the model generates a distribution of predictions. The entropy of this distribution serves as a measure of uncertainty; higher entropy signifies that the model is less certain about its output.
3.2 Calibration via Temperature Scaling
Neural networks are frequently overconfident, assigning high probabilities to incorrect results. To ensure that confidence scores are meaningful, the authors apply Temperature Scaling. This post-processing technique adjusts the logit outputs to align the model's confidence with its actual accuracy (e.g., a 0.7 confidence score should ideally represent a 70% probability of being correct).
3.3 Training Procedure
The training of AS-ConvQA follows a two-stage process:
- Stage 1: The model is initially trained on ground-truth (gold) data to establish a baseline of performance.
- Stage 2: The model undergoes fine-tuning where it is exposed to its own predicted answers. These answers are sampled based on the calibrated confidence and uncertainty scores, effectively simulating the noisy environment the model will encounter during inference.
4. Experimental Evaluation
The framework was evaluated using two major ConvQA benchmarks: QuAC (Question Answering in Context) and CoQA (Conversational Question Answering). For CoQA, the data was converted into an extractive format to ensure consistency with the QuAC dataset.
4.1 Results
AS-ConvQA demonstrated superior performance across different model architectures (BERT and RoBERTa):
- QuAC (BERT): AS-ConvQA achieved an F1 score of 57.35, compared to 55.76 for the "All Pred" baseline.
- CoQA (RoBERTa): The framework reached an F1 of 76.76, outperforming the "All Pred" score of 75.42.
4.2 Turn-based Analysis
The benefit of the filtering mechanism becomes increasingly apparent as the conversation progresses. In the QuAC dataset, the performance gap between AS-ConvQA and the baseline widened significantly by the third turn. While the baseline's accuracy degraded as errors accumulated, AS-ConvQA remained robust by preventing incorrect early answers from polluting the context of later questions.
5. Conclusion and Future Work
AS-ConvQA provides a practical solution to the problem of error propagation in realistic ConvQA settings. By calibrating confidence and measuring uncertainty, the model can effectively "know what it doesn't know."
However, limitations remain. The model can still be "certainly wrong," where it assigns high confidence to an incorrect answer, allowing it to pass the filter. Conversely, overly aggressive filtering can remove correct but low-confidence answers, causing the model to lose necessary context (such as pronoun references). Future research will focus on more sophisticated semantic detectors to better evaluate the relationship between conversational turns beyond simple probability metrics.
Summary updated