TL;DR
This paper presents the NCUEE-NLP system for the MEDIQA 2021 challenge, focusing on abstractive summarization of consumer health questions. By fine-tuning the pre-trained PEGASUS transformer model on a combination of existing and newly curated datasets, the system achieved a third-place ranking with a ROUGE2-F1 score of 0.1597. This work highlights the effectiveness of large pre-trained models for specialized medical text summarization tasks.
What Problem Does This Paper Address?
Consumers increasingly rely on online resources for health information, but health questions are often complex and difficult for automated systems to process. Summarizing these questions can significantly improve the performance of Question Answering (QA) systems. The MEDIQA 2021 challenge specifically targets this problem by focusing on abstractive summarization of consumer health questions, aiming to develop methods that can handle the unique challenges of long, complex, and often interrogative queries.
Methods
The NCUEE-NLP system leverages the PEGASUS transformer architecture, a powerful pre-trained model designed for abstractive summarization. PEGASUS utilizes a novel self-supervised objective called Gap Sentences Generation (GSG) alongside Masked Language Modeling (MLM) during its pre-training phase. For the MEDIQA 2021 challenge, the authors fine-tuned this pre-trained PEGASUS model on the specific task of consumer health question summarization. This fine-tuning process adapts the general summarization capabilities of PEGASUS to the nuances of medical queries.
Datasets
The system was trained and evaluated using a combination of datasets:
- MEDIQA 2021 Task Training Set: Provided by the challenge organizers.
- MEDIQA 2019 - NLI Dataset: Adapted by using entailment pairs where the longer sentence was treated as the question and the shorter as the summary.
- MEDIQA 2019 - RQE Dataset: Adapted by selecting question pairs that started with wh-words, be verbs, or auxiliary verbs, treating the longer question as the original and the shorter as the summary.
- MQP Dataset: Used similar question pairs, with longer questions as originals and shorter ones as summaries.
- EPIC-QA Dataset on COVID-19: Adapted by treating longer questions as originals and shorter ones as summaries.
Key Insights and Ideas
- Pre-trained models are powerful for specialized domains: The success of fine-tuning PEGASUS demonstrates that large, general-purpose pre-trained language models can be effectively adapted to niche domains like medical question summarization.
- Data augmentation is crucial: The authors' effort in collecting and processing additional datasets beyond the official training set was key to improving performance, especially given the relative scarcity of abstractive summarization datasets for interrogative sentences.
- XSum optimization helps: Models fine-tuned on the XSum dataset (which focuses on single-sentence summaries) generally outperformed their counterparts without this optimization, suggesting that this training objective aids in generating concise summaries.
Key Results
The NCUEE-NLP system achieved a ROUGE2-F1 score of 0.1597 on the MEDIQA 2021 consumer health question summarization task. This performance placed them third among 22 participating teams and ranked third out of 128 total submissions. The PEGASUS-Large-XSum model, in particular, showed strong performance across various metrics.
Limitations and Future Work
Not discussed in the paper.
Summary updated