NLM at MEDIQA 2021: Transfer Learning-based Approaches for Consumer Question and Multi-Answer Summarization

Shweta Yadav, Mourad Sarrouti, Deepak GuptaNAACL 2021

TL;DR

This paper tackles the challenge of summarizing complex consumer health questions and their corresponding answers, a crucial step for efficient health information retrieval. The authors leverage pre-trained transformer models, specifically employing transfer learning and incorporating medical entities to improve summarization quality for both question and multi-answer summarization tasks, achieving competitive results in the MEDIQA 2021 challenge.

What Problem Does This Paper Address?

The internet has become a primary source for health information, leading to a surge in consumer health questions. However, these questions are often verbose, containing peripheral details like medical history that aren't essential for retrieving precise answers. This complexity makes automatic answer retrieval challenging. To address this, the paper focuses on two key tasks: Question Summarization (QS), which aims to distill the core intent of a consumer's question, and Multi-Answer Summarization (MAS), which condenses multiple relevant answers into a concise summary. Efficiently summarizing these elements is vital for building robust and user-friendly health information systems.

Methods

The core of this work lies in applying transfer learning with pre-trained transformer models. For Question Summarization, the researchers experimented with ProphetNet, PEGASUS, and T5. They fine-tuned these models on task-specific datasets and introduced a novel approach involving medical entity extraction (using MetaMap and Scispacy) and medical entity coverage to filter and select the best question summaries. This ensures that the generated summaries retain crucial medical terms. For Multi-Answer Summarization, they employed an extractive method based on the T5 relevance-based re-ranking model. This involves splitting answers into sentences, ranking them by relevance to the question using T5, and then rejoining the top-ranked sentences to form the summary. Pre-processing steps like spelling correction and abbreviation expansion were also incorporated.

Datasets

The paper utilized several datasets for training and evaluation:

  • Question Summarization:
    • MeQSum: A dataset of 1,000 consumer health question-summary pairs created by medical experts.
  • Clinical Questions: A dataset of 4,655 clinical questions with short summaries.
  • MEDIQA-RQE: A dataset derived from consumer health questions and FAQs, used to create additional training pairs.
  • MedNLI: A dataset annotated by doctors for natural language inference, used to augment training data with entailed question pairs.
  • LiveQA17: A dataset containing 104 questions and their gold summaries.
  • Multi-Answer Summarization:
    • MS MARCO Passage: A large dataset for passage ranking.
  • MSMARCO MED: The medical subset of MS MARCO, focusing on medical queries.
  • MEDIQA-QA 2019: A dataset for medical question answering where answers were ranked by medical experts.

Key Insights and Ideas

  • Medical Entities are Crucial: The paper strongly emphasizes the importance of retaining medical entities in summarized questions. Their approach of extracting and ensuring coverage of these entities proved effective in selecting better summaries.
  • Transfer Learning Powerhouse: Pre-trained transformer models like T5, PEGASUS, and ProphetNet are highly effective for summarization tasks, especially when fine-tuned on relevant domain-specific data.
  • Data Augmentation Boosts Performance: Incorporating additional datasets (like MEDIQA-RQE, LiveQA17, and MedNLI) significantly improved the performance of the question summarization models, highlighting the benefit of diverse training data.
  • T5's Versatility: The T5 model demonstrated strong performance in both question summarization and multi-answer summarization, showcasing its adaptability across different summarization tasks.

Key Results

For the Question Summarization task, the authors achieved competitive results in the MEDIQA 2021 challenge:

  • Their best submission (Run-9) achieved the second-best ROUGE-1 score (35.58) among all submitted runs.
  • They also obtained strong scores for ROUGE-2 (15.14), HOLMS (56.59), and BERTScore (68.94).
  • Run-7 achieved the highest ROUGE-L score (31.16).

For the Multi-Answer Summarization task:

  • Their best run (Run-4) achieved ROUGE-1 (0.547), ROUGE-2 (0.468), and ROUGE-L (0.328).
  • Run-5 achieved the best BERTScore among their runs.
  • Run-1 achieved the highest HOLMS score.
  • The T5-based system was found to be competitive across various evaluation metrics.

Qualitatively, the paper noted that while pre-trained models generate readable and succinct summaries, further enhancements could be made by integrating information about question focus and types.

Limitations and Future Work

  • Limitations: The paper acknowledges that while T5 and PEGASUS generate factually correct summaries, they sometimes fail to produce fully correct summaries. The use of an open-domain dataset (Quora) for question summarization did not improve performance, suggesting it might not be well-suited for the medical domain.
  • Future Work: The authors plan to explore techniques to better integrate medical entities and semantics into pre-trained transformer models for question summarization. They also intend to investigate abstractive approaches for multi-answer summarization.

Summary updated

BibTeX
@inproceedings{yadav2021nlm,
  author = {Shweta Yadav and Mourad Sarrouti and Deepak Gupta},
  title = {{NLM at MEDIQA 2021: Transfer Learning-based Approaches for Consumer Question and Multi-Answer Summarization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure