SCAI-QReCC Shared Task on Conversational Question Answering

Svitlana Vakulenko, Johannes Kiesel, Maik FröbearXiv 2022 · arXiv

TL;DR

The SCAI-QReCC shared task tackles the difficulty of evaluating generative models in conversational question answering (ConvQA). By introducing a two-stage human evaluation framework that distinguishes between answer plausibility and faithfulness, the authors identified significant "hallucination" issues in state-of-the-art models and extended the QReCC dataset with a diverse set of alternative correct answers.

What Problem Does This Paper Address?

Conversational Question Answering (ConvQA) requires systems to understand follow-up questions that are often ambiguous without context (e.g., "How much do they make?" following a discussion about nurses). While generative models can produce fluent, natural-sounding responses, evaluating them is a nightmare.

Standard automatic metrics like ROUGE or F1 rely on token overlap with a single "ground truth" answer, which fails in two ways:

  1. False Negatives: A system might provide a perfectly correct answer that uses different phrasing or focuses on a different factual aspect than the ground truth.
  2. False Positives: Generative models often "hallucinate"—producing answers that look plausible to a human but are factually unsupported by the source documents.

The SCAI-QReCC task sought to benchmark systems on the large-scale QReCC dataset and establish a more robust evaluation methodology to bridge this research gap.

Methods

The task followed a standard decomposition of the ConvQA pipeline:

  1. Question Rewriting: Transforming a conversational turn into a standalone, unambiguous query.
  2. Passage Retrieval: Locating relevant information from a collection of web snippets.
  3. Answer Generation: Producing a faithful, grammatical response based on retrieved evidence.

To evaluate these systems, the authors moved beyond simple overlap metrics. They implemented a two-stage human evaluation pipeline (illustrated in Figure 3):

  • Stage 1: Plausibility. Crowd workers judged if a generated answer appeared to answer the question, regardless of its factual accuracy.
  • Stage 2: Faithfulness. For plausible answers, the authors used a token-overlap heuristic and Semantic Answer Similarity (SAS) to extract short "evidence spans" from the retrieved passages. Humans then judged if the answer was actually supported by these specific spans.
Figure 3: Our human evaluation including assessment of the answer plausibility and answer faithfulness.

Datasets

The shared task centered on QReCC, a dataset containing ~14k conversational sequences (81k QA pairs) where answers are grounded in web pages.

A major outcome of this work is an extended version of the QReCC test set. By manually verifying 1,863 plausible answers from participant runs, the authors added hundreds of alternative correct answers to the dataset, making it a more robust benchmark for future research.

Key Insights and Ideas

  • The "Plausibility Trap": Models, particularly large language models like GPT-3 (used as a baseline), are excellent at generating answers that look right but are factually incorrect. As shown in Figure 1, a model might confidently state a salary figure that is completely fabricated or outdated compared to the provided evidence.
  • Retrieval is the Bottleneck: There is a massive performance gap between using automatically rewritten questions versus human-rewritten ones for retrieval. Figure 2 highlights that even simple baselines perform significantly better when the input query is perfectly disambiguated.
  • Metric Disagreement: The authors found that different automatic metrics (BERTScore, SAS, F1) often disagree on system rankings. This underscores the necessity of human-in-the-loop evaluation for generative tasks.
Figure 1: A sample from the SCAI-QReCC results with the original conversational question, disambiguated rewritten question, a ground-truth human answer, an unfaithful generated answer from one of the participat- ing systems and an alternative correct answer extracted from the submitted passages using our approach.

Key Results

  • Top Performers: Team Rachael achieved the highest scores across most metrics by using a T5-based rewriter and a PEGASUS-based abstractive summarizer for generation.
  • Dense Retrieval Challenges: Interestingly, none of the participants successfully deployed dense passage retrieval (like DPR) at scale for this task, likely due to the large size of the QReCC collection. All teams relied on sparse BM25 retrieval, sometimes paired with BERT-based re-rankers.
  • Human Eval Findings: Out of 1,863 answers judged as "plausible," only 386 were confirmed as "faithful" to the evidence. This highlights a high rate of hallucination or retrieval failure across all submitted systems.
Figure 2: Highest scores reached per dataset variant for each metric. For all metrics except POSS a score of 1 corresponds to optimal performance.

Limitations and Future Work

The authors acknowledge that their evidence-sampling approach (using SAS to find spans) might miss relevant evidence if the generated answer is phrased very differently from the source text.

For future shared tasks, they propose a new requirement: Generative models should be required to output short evidence spans (citations) alongside their answers. This would make human verification of faithfulness much more efficient and reliable, moving the field toward more trustworthy conversational AI.

Summary updated

Abstract

Search-Oriented Conversational AI (SCAI) is an established venue that regularly puts a spotlight upon the recent work advancing the field of conversational search. SCAI'21 was organised as an independent on-line event and featured a shared task on conversational question answering. Since all of the participant teams experimented with answer generation models for this task, we identified evaluation of answer correctness in this settings as the major challenge and a current research gap. Alongside the automatic evaluation, we conducted two crowdsourcing experiments to collect annotations for answer plausibility and faithfulness. As a result of this shared task, the original conversational QA dataset used for evaluation was further extended with alternative correct answers produced by the participant systems.
BibTeX
@misc{vakulenko2022scai,
  author = {Svitlana Vakulenko and Johannes Kiesel and Maik Fröbe},
  title = {{SCAI-QReCC Shared Task on Conversational Question Answering}},
  year = {2022},
  eprint = {2201.11094},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.11094},
}
Figure