TL;DR
This paper introduces a novel framework for automatically generating biomedical question-answering (QA) datasets from online Q&A forums. They created the BiQA corpus, a valuable resource for training and evaluating QA systems, by extracting question-article pairs from forums like Stack Exchange and Reddit, addressing the scarcity of real-world biomedical QA data.
What Problem Does This Paper Address?
Developing effective Question Answering (QA) systems, especially in specialized domains like biomedicine, is challenging due to the complexity of the subject matter and the lack of large, diverse training datasets. Existing datasets often rely on expert-annotated questions, which may not fully capture the variability and nuances of real user queries. This gap hinders the development of QA systems that can accurately retrieve relevant information from the vast amount of biomedical literature to answer user questions.
Methods
The core of this work is a framework designed to automatically extract question-article pairs from Q&A forums. The process involves:
- Forum Selection: Identifying Q&A forums relevant to the biomedical domain (e.g., Biology.SE, Medical Sciences.SE, r/nutrition).
- Question and Answer Extraction: Retrieving posts, specifically those with question marks, and their first-level replies (answers).
- Hyperlink Extraction and Mapping: Parsing answers to find hyperlinks and mapping them to PubMed IDs (PMIDs) using various URL types (PMC, doi.org, etc.). This is a crucial step for linking user-provided answers to scientific articles.
- Corpus Generation: Assembling the extracted question-article pairs into the BiQA corpus.
The authors also explored various baseline document retrieval methods (NCBI Eutils API, query likelihood, BM25) to evaluate the quality of their generated dataset and its suitability for training QA systems.
Datasets
The primary contribution is the BiQA corpus, generated from three Q&A forums:
- Biology.SE: A Stack Exchange forum focused on biology.
- Medical Sciences.SE: A Stack Exchange forum for medical sciences.
- r/nutrition: A Reddit subreddit dedicated to nutrition.
The BiQA corpus consists of 7,453 questions and 14,239 question-article pairs. This corpus differs from existing ones like BioASQ by being based on real user questions and evolving over time, offering a more realistic benchmark for QA systems.
Key Insights and Ideas
- Q&A Forums as a Rich Source: Online Q&A forums are a valuable, underutilized resource for generating realistic, user-driven QA datasets, especially for complex domains like biomedicine.
- User-Provided Citations are Key: The practice of users citing scientific articles in their answers provides a direct link to relevant documents, enabling automatic corpus generation.
- Real-World Questions are Diverse: Questions from forums exhibit a wider range of complexity and phrasing compared to expert-curated datasets, making them more challenging but also more representative of actual user needs.
- Article Similarity to Questions: The articles cited in answers are significantly more similar to the questions than random articles, validating the relevance of the extracted pairs.
Key Results
- The framework successfully generated the BiQA corpus with 7,453 questions and 14,239 question-article pairs.
- Baseline document retrieval methods showed that articles cited in answers are significantly more similar to the questions than random articles (p-value < 0.001).
- A state-of-the-art deep learning model trained on the BiQA corpus achieved performance comparable to models trained on expert-annotated datasets.
- Adding the BiQA corpus to the BioASQ6b training set led to marginal improvements in QA system performance on most test batches.
- The study found that for some datasets (Biology.SE), increasing the minimum number of cited PMIDs per answer significantly improved MAP scores, suggesting a trade-off between corpus size and retrieval quality.
Limitations and Future Work
- Anonymity and Bias: Q&A forums are anonymous, which can lead to personal biases in answers and a lack of accountability.
- Context of Citations: The framework does not capture the context or ranking of citations within an answer, meaning it treats all cited articles equally, even if some are more relevant than others.
- Community Engagement: The quality and quantity of extracted data can depend on the engagement levels of users within a specific forum.
- Future Work: The authors suggest using semantic representations (word embeddings) to find similar questions and explore other Q&A forums or social media platforms. They also propose that expert curation could further enhance the dataset's effectiveness. The framework is designed to be extensible, allowing for updates as new posts are made.
Summary updated