Accurate Training of Web-based Question Answering Systems with Feedback from Ranked Users

Liang Wang, Ivano Lauriola, Alessandro MoschittiACL 2023 · DOI

TL;DR

This paper tackles the challenge of training accurate Question Answering (QA) systems using noisy, real-world user feedback. They introduce two novel methods, "Relevance Filtering" and "Collaborative Filtering," to identify and leverage reliable user feedback, significantly improving QA model performance compared to using raw, unfiltered data.

What Problem Does This Paper Address?

Training state-of-the-art Question Answering (QA) systems typically requires massive, meticulously annotated datasets. Creating these datasets is expensive and time-consuming. While deployed QA systems, like virtual assistants, generate a wealth of user feedback for free, this feedback is inherently noisy. Users are not expert annotators and their judgments can be subjective, inconsistent, or even incorrect. Previous attempts to use this noisy feedback often relied on artificial noise models that don't reflect real-world user behavior, or they failed to effectively aggregate and filter this data. This paper aims to bridge this gap by developing methods to effectively filter and utilize real-world user feedback to train more accurate QA models.

Methods

The core of this paper lies in its two proposed methods for filtering noisy user feedback:

  • Relevance Filtering: This approach leverages the agreement between a QA system's predicted answer score and the user's feedback. The intuition is that if a user consistently agrees with the model's assessment (or disagrees when the model is likely wrong), their feedback is more reliable. They first measure the agreement between a pre-trained answer selection model's score and the user's binary feedback for each question-answer pair. Then, they compute a "Relevance Score" (RS) for each user by averaging these agreement scores across all their feedback instances. Users with higher RS are considered more reliable.
  • Collaborative Filtering: This method assumes that users have different levels of expertise for different types of questions. It clusters similar questions together and then analyzes how individual users' feedback aligns with the majority feedback within those clusters. If a user consistently deviates from the consensus, their feedback is deemed less reliable. They represent user-question interactions in a matrix and compute a "voted feedback" for each question. A user's reliability is then measured by their average proximity to this voted feedback. To handle the sparsity of user-question interactions, they extend this by clustering semantically similar questions, creating user-cluster interaction matrices.

Datasets

  • Internal Feedback Dataset (IFD): A large-scale dataset of 16 million question-answer-feedback tuples collected from the traffic of a popular virtual assistant. This dataset is the primary source of real-world noisy feedback.
  • ASNQ (Answer Sentence Selection Natural Questions): A large-scale, high-quality annotated dataset derived from Natural Questions, used for pre-training and fine-tuning models.
  • WikiQA: A smaller, in-domain dataset used as the target domain for evaluating the trained QA models.

Key Insights and Ideas

  • Real-world noise is patterned: Unlike previous work that used uniform random noise, this paper highlights that real user feedback noise follows specific patterns influenced by user behavior and question characteristics.
  • User reliability is key: Not all users are equally good annotators. Identifying and leveraging the feedback from more reliable users is crucial for improving model performance.
  • Agreement is a signal of reliability: The "Relevance Filtering" method demonstrates that when a user's feedback aligns with a well-performing QA model's predictions, it's a strong indicator of that user's reliability.
  • Collaborative consensus matters: The "Collaborative Filtering" method shows that users who consistently disagree with the majority feedback on similar questions are likely less reliable.
  • Even noisy data can be valuable: Despite its inherent noise, the large-scale IFD dataset, when filtered, significantly improves QA model performance, even when models are already trained on high-quality data.

Key Results

  • Improved performance with filtered data: Both Relevance Filtering and Collaborative Filtering significantly improve QA model performance compared to using raw, unfiltered IFD data.
  • Relevance Filtering outperforms: Manual evaluation and model training results suggest that Relevance Filtering is more effective at ranking tuples and achieving better final performance than Collaborative Filtering.
  • Top 10% is often sufficient: Using only the top 10% of feedback ranked by Relevance Filtering can significantly reduce data size while improving performance.
  • Sequential fine-tuning is beneficial: Training models sequentially (e.g., ASNQ -> IFD -> WikiQA) consistently improves performance, demonstrating the value of incorporating filtered feedback even after training on large, high-quality datasets.
  • Table 3 shows concrete improvements: For example, using Relevance Filtering's "Top" 10% data on WikiQA achieves a P@1 of 87.0, MAP of 90.7, and MRR of 92.0, outperforming the "Full" IFD baseline.

Limitations and Future Work

  • Relevance Filtering bias: The authors acknowledge that Relevance Filtering might introduce a selection bias, potentially penalizing uncertain or boundary-case questions that are close to the model's classification boundary.
  • Clustering in Collaborative Filtering: The clustering strategy in Collaborative Filtering can introduce noise by aggregating non-equivalent questions, potentially reducing the reliability of the voted feedback vector.
  • Data privacy: The IFD dataset, being real user data from a commercial virtual assistant, cannot be publicly released, limiting direct replicability.
  • Future work: The authors suggest further improvements to Collaborative Filtering by exploring different clustering approaches and embedding extractors. They also highlight the importance of this work as a first step in analyzing and integrating real feedback data into QA model training, opening promising directions for future research.

Summary updated

BibTeX
@inproceedings{wang2023accurate,
  author = {Liang Wang and Ivano Lauriola and Alessandro Moschitti},
  title = {{Accurate Training of Web-based Question Answering Systems with Feedback from Ranked Users}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-industry.63},
}
Figure