Improving Health Question Answering with Reliable and Time-Aware Evidence Retrieval

Juraj Vladika, Florian MatthesNAACL 2024 · DOI

TL;DR

This paper tackles the challenge of building reliable health question-answering (QA) systems by focusing on how to effectively retrieve relevant evidence from a massive corpus of biomedical literature. They found that prioritizing recent and highly cited research papers significantly improves answer accuracy, suggesting that "time-aware" retrieval is crucial for trustworthy health information.

What Problem Does This Paper Address?

In the digital age, people increasingly turn to the internet for health information. While question-answering (QA) systems can automate this process, existing ones often struggle with "novel" questions because they rely on pre-selected evidence. This is particularly problematic in the medical field, where clinical recommendations and understanding of diseases evolve rapidly. The core challenge is to build open-domain QA systems that can sift through vast amounts of information (like millions of research abstracts) to find the most accurate and up-to-date evidence to answer a user's health query. Simply retrieving a large number of documents isn't enough; the quality and relevance of that evidence, especially its timeliness, are paramount.

Methods

The researchers employed a standard "retrieve-then-read" QA pipeline. The key innovation lies in their experimental manipulation of the retrieval component. They used BM25 for initial document retrieval and a sentence embedding model (SPICED) for sentence selection. Crucially, they didn't just retrieve documents; they experimented with:

  • Varying the number of retrieved documents (top k): How many documents should the system consider?
  • Varying the number of retrieved sentences (top j): Should the system focus on entire abstracts or just key sentences?
  • Filtering by evidence quality: This involved using metadata like the publication year and the number of citations to prioritize more recent and influential research.

The "reader" component (a DeBERTa-v3 model fine-tuned for Natural Language Inference) remained constant across experiments to isolate the impact of retrieval strategies.

Datasets

The study utilized three diverse datasets for evaluating their health QA system:

  • HealthFC-3: A dataset of consumer health questions with "supported," "refuted," and "not enough information" labels.
  • HealthFC-2: A binary version of HealthFC (supported/refuted).
  • BioASQ-7b: A dataset designed for biomedical experts, featuring more complex research questions with "supported" and "refuted" labels.
  • TREC-Health: A dataset focused on common consumer health concerns, also with "supported" and "refuted" labels.

The evidence corpus was a massive collection of 20.6 million biomedical research abstracts from PubMed, updated to 2022.

Key Insights and Ideas

  • Less is More (for Documents): Retrieving fewer documents (specifically, the top 1-5) generally led to better performance than retrieving a large number (50-100). This suggests that too much unfiltered evidence can overwhelm the reader model and introduce noise.
  • Recency Matters: More recent research papers consistently led to more accurate answers. This highlights the dynamic nature of medical knowledge and the importance of up-to-date information.
  • Citations as a Proxy for Quality: Highly cited papers also correlated with better performance, indicating that influential research is often more reliable.
  • Time-Awareness is Key: The combination of recency and citation count proved to be a powerful strategy for improving the reliability of health QA.
  • Evidence Disagreement is a Real Problem: The paper acknowledges that different studies can offer conflicting conclusions, and their majority-voting approach simplifies this complexity.

Key Results

  • Improved F1 Scores: By limiting retrieved documents to the top 1-5, the F1 score improved by up to 18% for binary datasets (HealthFC-2, TREC-Health) compared to retrieving 100 documents.
  • Time-Awareness Boost: Filtering evidence by publication year and citation count showed a clear trend of improved performance, with more recent and highly cited papers yielding better results. For instance, limiting to papers published after 2020 on HealthFC-2 improved F1 scores.
  • Sentence vs. Document: The impact of using only sentences versus full documents was less consistent. For some datasets, more sentences improved performance, while for others, fewer sentences were better, suggesting a complex interplay.

Limitations and Future Work

The authors acknowledge several limitations:

  • Model Complexity: The QA pipeline involves multiple components, and isolating the exact cause of incorrect predictions can be challenging.
  • Evidence Disagreement: The majority voting strategy for handling conflicting evidence simplifies the problem but loses information about prediction uncertainty.
  • Interpretability: The current system predicts an answer but doesn't explain why, which is crucial for end-users.

Future work directions include:

  • Strength of Evidence: Incorporating more nuanced measures of evidence strength, such as study type (e.g., systematic review vs. single study), using frameworks like GRADE.
  • Handling Disagreement: Developing methods to better represent and utilize information about conflicting evidence and label variation.
  • Interpretability and User-Centricity: Enhancing the system to provide explanations for its predictions, potentially using large language models (LLMs).

Summary updated

BibTeX
@inproceedings{vladika2024improving,
  author = {Juraj Vladika and Florian Matthes},
  title = {{Improving Health Question Answering with Reliable and Time-Aware Evidence Retrieval}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.295},
}
Figure