Generating Scientific Question Answering Corpora from Q&A forums

Andre Lamurias, Diana F. Sousa, Francisco M. Couto

Abstract

Question Answering (QA) is a natural language processing task that aims at obtaining relevant answers to user questions. While some progress has been made in this area, biomedical questions are still a challenge to most QA approaches, due to the complexity of the domain and limited availability of training sets. We present a method to automatically extract question-article pairs from Q&A web forums, which can be used for document retrieval, a crucial step of most QA systems. The proposed framework extracts from selected forums the questions and the respective answers that contain citations. This way, QA systems based on document retrieval can be developed and evaluated using the question-article pairs annotated by users of these forums. We generated the BiQA corpus by applying our framework to three forums, obtaining 7,453 questions and 14,239 question-article pairs. We evaluated how the number of articles associated with each question and the number of votes on each answer affects the performance of baseline document retrieval approaches. Also, we demonstrated that the articles given as answers are significantly similar to the questions and trained a state-of-the-art deep learning model that obtained similar performance to using a dataset manually annotated by experts. The proposed framework can be used to update the BiQA corpus from the same forums as new posts are made, and from other forums that support their answers with documents.

Introduction

Biomedical Question Answering (QA) is a high-stakes field where researchers and clinicians require precise, evidence-based information. However, developing AI systems for this domain is hindered by the "Expert Bottleneck." Traditional benchmarks, such as BioASQ, rely on domain experts to manually curate questions and link them to scientific literature. This process is inherently slow, expensive, and often fails to reflect the natural language queries used by real-world practitioners. There remains a significant gap between the structured questions in academic datasets and the complex, varied queries found in practice.

Methodology

To address the scarcity of training data, we propose a framework that leverages the "wisdom of the crowd" from popular Q&A forums. By treating these forums as a source of "silver standard" data, we can generate large-scale corpora without manual annotation.

The BiQA corpus generation pipeline, from forum scraping to Q-PubMed pair creation.

As illustrated in the pipeline above, the process follows three primary stages:

  1. Extraction: Data is scraped from targeted forums, specifically Biology Stack Exchange, Medical Sciences Stack Exchange, and the r/nutrition subreddit.
  2. Citation Mapping: The framework identifies hyperlinks within forum answers (e.g., PMC, DOI, ResearchGate) and automatically maps them to PubMed IDs (PMIDs).
  3. Pairing: When a question is linked to an answer containing a scientific citation, a Question-Article pair is formed.

Evaluation Framework

To validate the utility of the generated BiQA corpus, we implemented an evaluation pipeline to test document retrieval performance. This involves processing the text from titles, bodies, and answers to retrieve relevant documents from PubMed using various ranking functions.

The evaluation pipeline used to assess document retrieval performance on the BiQA corpus.

Key Insights and Results

Our analysis of the BiQA corpus, which contains 7,453 questions and 14,239 question-article pairs, revealed several critical findings:

  • The Vocabulary Gap: Standard retrieval models (like BM25) struggle to find correct articles using only the user's question. However, performance improves significantly when using the answer text as the query, highlighting the linguistic distance between layperson queries and scientific literature.
  • Crowds vs. Experts: A state-of-the-art deep learning model (TERM-PACRR) trained on the automatically generated BiQA data achieved performance levels remarkably close to models trained on expert-curated BioASQ data. In several test batches, the difference in Mean Average Precision (MAP) was less than 0.01.
  • The Vote Paradox: Interestingly, filtering data by the number of citations per answer proved more effective for dataset quality than filtering by user upvotes. On the Biology.SE subset, the highest MAP (0.2019) was reached in answers containing at least 10 PMIDs.
  • Model Boosting: Integrating BiQA data into the standard BioASQ training set improved deep learning model performance across 4 out of 5 test batches, demonstrating its value as a supplemental training resource.

Limitations and Future Work

While the BiQA framework provides a scalable solution for data generation, it faces challenges such as contextual noise—where citations may be tangential rather than direct answers—and potential biases inherent in anonymous crowdsourced content. Furthermore, the current system does not rank the relative relevance of multiple citations within a single answer.

Future research will focus on utilizing semantic representations (word embeddings) to improve question-document matching and expanding the framework to additional platforms like Quora and social media to further diversify the corpus.

Summary updated

Abstract

Question Answering (QA) is a natural language processing task that aims at obtaining relevant answers to user questions. While some progress has been made in this area, biomedical questions are still a challenge to most QA approaches, due to the complexity of the domain and limited availability of training sets. We present a method to automatically extract question-article pairs from Q\&A web forums, which can be used for document retrieval, a crucial step of most QA systems. The proposed framework extracts from selected forums the questions and the respective answers that contain citations. This way, QA systems based on document retrieval can be developed and evaluated using the question-article pairs annotated by users of these forums. We generated the BiQA corpus by applying our framework to three forums, obtaining 7,453 questions and 14,239 question-article pairs. We evaluated how the number of articles associated with each question and the number of votes on each answer affects the performance of baseline document retrieval approaches. Also, we demonstrated that the articles given as answers are significantly similar to the questions and trained a state-of-the-art deep learning model that obtained similar performance to using a dataset manually annotated by experts. The proposed framework can be used to update the BiQA corpus from the same forums as new posts are made, and from other forums that support their answers with documents. The BiQA corpus and the framework used to generate it are available at \url{https://github.com/lasigeBioTM/BiQA}.
BibTeX
@inproceedings{lamurias2020generating,
  author = {Andre Lamurias and Diana F. Sousa and Francisco M. Couto},
  title = {{Generating Scientific Question Answering Corpora from Q\&A forums}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=1VMFoiYpm8},
}
Figure