WBI at MEDIQA 2021: Summarizing Consumer Health Questions with Generative Transformers

Mario Sänger, Leon Weber, Ulf LeserNAACL 2021

TL;DR

This paper tackles the challenge of summarizing lengthy consumer health questions for better question answering systems. The authors fine-tuned large generative transformer models (PEGASUS and BART) and explored three extensions: incorporating structured knowledge, using a generator-discriminator ensemble, and disentangling summarization from interrogative prediction. Their best model, a fine-tuned PEGASUS, achieved second place in the MEDIQA 2021 competition, though surprisingly, the extensions didn't improve performance on the official test set, highlighting the complexities of evaluating summarization models.

What Problem Does This Paper Address?

The internet is a vast repository of health information, and consumers increasingly turn to it to answer their medical questions. However, these questions are often long and contain extraneous details, making it difficult for search engines and QA systems to pinpoint the core concern. This necessitates automatic summarization techniques that can distill these complex questions into concise, actionable queries. The MEDIQA 2021 shared task specifically focused on this problem, aiming to develop better summarization methods for consumer health questions.

Methods

The core of the approach involves fine-tuning large pre-trained generative transformer models, specifically PEGASUS and BART, for the task of conditional generation. This means the model takes the original question as input and generates a summary. They explored three main extensions to improve upon this baseline:

  • Integration of Structured Knowledge via Entity Embeddings: Recognizing that health questions often revolve around specific entities (diseases, medications), they enriched the input embeddings of the transformer models with named entity recognition (NER) information. This was done using domain-specific NER models like HunFlair and a custom Flair model trained on the CHQA corpus.
  • Ensembling Multiple Generative Transformers: Instead of simply averaging outputs, they employed a generator-discriminator framework. Multiple generative models produce candidate summaries, and a BERT-based discriminator selects the most promising one based on predicted ROUGE-L-F1 scores.
  • Disentangling Summarization and Interrogative Prediction: Observing that question categories often align with the interrogative form of the summary (e.g., "What is...?", "How can...?"), they developed a separate BERT-based classification model to predict the interrogative. This predicted interrogative then guides the summarization process.

Datasets

  • MeQSum Corpus: Used for training the models. It contains 1,000 consumer health questions and their summaries.
  • MEDIQA 2021 Shared Task Data: Official validation and test sets were used for evaluation. The paper also describes a post-hoc evaluation setting that combined MeQSum and the shared task validation data to create larger training and validation splits (850 and 200 questions, respectively) to mitigate issues with small dataset sizes.

Key Insights and Ideas

  • Vanilla PEGASUS is a Strong Baseline: The fine-tuned PEGASUS model, without any extensions, achieved a very competitive ROUGE-2-F1 score, demonstrating the power of large pre-trained models for this task.
  • Extensions Hurt Performance on Official Test Set: Surprisingly, all three proposed extensions (entity embeddings, generator-discriminator, interrogative prediction) led to a decrease in performance on the official MEDIQA 2021 test set. This suggests that either the extensions were not well-suited for the specific test data, or the evaluation metric (ROUGE) might not fully capture the benefits of these more complex approaches.
  • Post-Hoc Analysis Shows Promise: When evaluated on a larger, custom validation set (Setting 2), the extensions did show slight performance improvements. This indicates that the small size of the official validation and test sets might have masked the true benefits of the proposed methods.
  • ROUGE Metric Limitations: The authors acknowledge that ROUGE has limitations in evaluating summaries, particularly concerning synonyms, abbreviations, and enumerations, which can affect the interpretation of results.
  • Interrogative Prediction Bias: The interrogative prediction models showed a bias towards majority classes, predicting common interrogative forms even when not appropriate for the specific question.

Key Results

  • Second Place in MEDIQA 2021: The authors' best performing model, a fine-tuned vanilla PEGASUS, achieved a ROUGE-2-F1 score of 15.99, securing second place in the competition.
  • Extensions' Impact: On the official test set (Setting 1), the extensions generally hurt performance. For instance, the generator-discriminator ensemble (Run 7) achieved 14.6 ROUGE-2-F1, while the best baseline was 16.0.
  • Setting 2 Improvements: In the post-hoc evaluation (Setting 2), the best BART-large model achieved ROUGE-2-F1 of 34.06. Entity embeddings improved this to 34.5, and the interrogative prediction approach with BioMed-ROBERTa achieved 33.71 (though slightly lower than the BART-large baseline in this setting).
  • BERTScore Performance: One of the generator-discriminator runs (Run 8) achieved the top ranking under the BERTScore metric, suggesting that different evaluation metrics can highlight different strengths of the models.

Limitations and Future Work

  • Small Evaluation Sets: The primary limitation acknowledged is the small size of the official validation and test sets, which leads to high variance in results and makes drawing definitive conclusions difficult.
  • ROUGE Metric: The inherent limitations of the ROUGE metric for evaluating generated summaries are noted as a factor that might obscure the true performance of the models.
  • Interrogative Prediction Bias: Future work could focus on mitigating the bias observed in the interrogative prediction models.
  • Discriminator Improvement: For the generator-discriminator ensemble, improving the discriminator's ability to select the best summary is identified as a key area for future research.
  • Further Investigation Needed: The authors conclude that further investigation is necessary to draw definitive conclusions about the effectiveness of their proposed modifications.

Summary updated

BibTeX
@inproceedings{sanger2021wbi,
  author = {Mario Sänger and Leon Weber and Ulf Leser},
  title = {{WBI at MEDIQA 2021: Summarizing Consumer Health Questions with Generative Transformers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure