Answer Quality Aware Aggregation for Extractive QA Crowdsourcing

Peide Zhu, Zhen Wang, Claudia Hauff, Jie Yang, Avishek Anand

Introduction

Building high-quality extractive question-answering (EQA) datasets, such as SQuAD or NewsQA, typically relies on crowdsourcing. In these tasks, multiple workers are asked to identify specific answer spans within a provided text. However, a significant challenge arises: workers frequently disagree on exact span boundaries or provide incorrect answers.

Standard aggregation techniques like Majority Voting often fail in EQA because the "label space"—representing every possible start and end position in a document—is massive. This makes reaching a natural consensus rare. While automated text aggregation methods exist, they often treat answers as isolated strings, ignoring the vital relationship between the question, the answer span, and the surrounding context. This research introduces a "quality-aware" aggregation model to bridge this gap.

The ACAF Framework

The authors propose ACAF (Answer Quality Aware Aggregation), a framework that enhances traditional sequence aggregation by incorporating two specific quality metrics derived from pre-trained language models (PLMs).

The ACAF framework architecture, illustrating the flow from raw annotations to quality-weighted aggregation using NLI and QA signals.

As shown in the architecture above, the framework utilizes two primary signals to evaluate worker contributions:

  1. Answer Confidence (AF): A pre-trained BERT-QA model calculates the geometric mean of the start and end position probabilities for a given worker's span. This metric essentially measures how likely a standard QA model thinks a specific span is the correct answer.
  2. Answer Correctness (AC): This treats the problem as a Natural Language Inference (NLI) task. The passage serves as the premise, and the question-answer pair is reformulated into a hypothesis. The model predicts the probability that the passage entails the QA pair.

These scores are combined with semantic embeddings to weight worker answers during aggregation. Instead of simply seeking the most common answer, the framework identifies a "centroid" answer that is both semantically representative of the group and rated highly by the AF and AC models.

Key Insights: The Power of Complementary Signals

A core finding of the study is that QA confidence (AF) and NLI-based verification (AC) capture different types of errors. While AF is effective at identifying spans that look plausible to a QA model, AC is superior at verifying if the span logically answers the question based on the text.

ROC curves showing that the combination of Answer Correctness (AC) and Answer Confidence (AF) provides a stronger signal for identifying correct answers than either metric alone.

As demonstrated in the ROC analysis, combining these metrics (AC+AF) consistently outperforms using either in isolation. Furthermore, the authors found that "Context-Free" aggregation performs poorly; using contextual embeddings from PLMs is vital for understanding the nuances of worker selections.

Experimental Results

The ACAF framework was evaluated primarily on NewsQA, a dataset containing raw, unaggregated annotations from multiple workers, including difficult non-consensus cases. Additional validation was performed on SQuAD and Natural Questions.

Key performance highlights include:

  • Precision Boost: ACAF methods outperformed baseline "Random Selection" and "Context-Free" models by approximately 16% in Precision@1.
  • Non-Consensus Performance: On the NewsQA "Test-NC" (non-consensus) set, the ACAF-SMS method achieved a P@1 of 0.6304, compared to 0.4782 for random selection.
  • Downstream Utility: QA models trained on answers aggregated by ACAF performed nearly as well as those trained on human-validated "gold" labels. This suggests the method could significantly reduce the need for expensive secondary human verification rounds.

Limitations and Future Work

While effective, the current model treats all workers as anonymous and does not account for individual worker reputation or historical accuracy. Additionally, the framework assumes a single "true" correct answer span exists for each question, whereas some questions may have multiple valid spans. Future research directions include exploring multi-answer settings and integrating worker reliability scores to further refine the aggregation process.

Summary updated

Abstract

Quality control is essential for creating extractive question answering (EQA) datasets via crowdsourcing. Aggregation across answers, i.e. word spans within passages annotated, by different crowd workers is one major focus for ensuring its quality. However, crowd workers cannot reach a consensus on a considerable portion of questions. We introduce a simple yet effective answer aggregation method that takes into account the relations among the answer, question, and context passage. We evaluate answer quality from both the view of question answering model to determine how confident the QA model is about each answer and the view of the answer verification model to determine whether the answer is correct. Then we compute aggregation scores with each answer’s quality and its contextual embedding produced by pre-trained language models. The experiments on a large real crowdsourced EQA dataset show that our framework outperforms baselines by around 16% on precision and effectively conduct answer aggregation for extractive QA task.
BibTeX
@inproceedings{zhu2022answer,
  author = {Peide Zhu and Zhen Wang and Claudia Hauff and Jie Yang and Avishek Anand},
  title = {{Answer Quality Aware Aggregation for Extractive QA Crowdsourcing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.457},
}
Figure