UETfishes at MEDIQA 2021: Standing-on-the-Shoulders-of-Giants Model for Abstractive Multi-answer Summarization

Hoang-Quynh Le, Quoc-An Nguyen, Quoc-Hung Duong, Minh-Quang Nguyen, Huy-Son Nguyen, Tam Doan Thanh, Hai-Yen Thi Vuong, Trang M. NguyenNAACL 2021

TL;DR

This paper introduces a novel abstractive summarization model, "Standing-on-the-Shoulders-of-Giants" (SSG), designed for the challenging MEDIQA 2021 task of summarizing multiple answers to consumer health questions. By leveraging the power of BART and incorporating innovative query-driven filtering mechanisms, the SSG model achieved impressive results, ranking 2nd and 3rd in the competition.

What Problem Does This Paper Address?

In the era of big data, accessing and understanding vast amounts of biomedical information, especially for consumers, is a significant challenge. While search engines and question-answering systems help retrieve information, the sheer volume and complexity of unstructured text often make it difficult for non-experts to discern correct and relevant content. Automatic summarization aims to tackle this by generating concise descriptions of salient information. However, summarizing multiple answers to a single question, a task known as multi-answer summarization, presents a unique challenge: it requires both aggregating information from various sources and generating a coherent, concise summary. The MEDIQA 2021 shared task specifically focused on this problem in the medical domain, aiming to advance approaches that can effectively handle the aggregation and summarization of multiple relevant answers.

Methods

The core of the UETfishes system is an abstractive summarization model built upon BART (Bidirectional and Auto-Regressive Transformer), a powerful pre-trained sequence-to-sequence model. The authors' key innovation lies in their "Standing-on-the-Shoulders-of-Giants" (SSG) approach, which enhances BART's performance by introducing two crucial filtering phases designed to make the input more question-oriented and concise.

  1. Pre-processing: This initial step cleans the input data by removing HTML tags, non-UTF-8 characters, and redundant formatting. It also utilizes scispaCy for typical natural language processing tasks like segmentation and tokenization.
  2. Coarse-grained Filtering: This phase aims to remove sentences that are less relevant to the question. It employs a "query-driven" strategy, calculating the cosine similarity between BioBERT embeddings of the question and each sentence. Two main methods are used:
  3. Top-n query-driven sentences: Selects sentences with the highest similarity scores.
  4. Top-n query-driven passages: Identifies passages that are likely to contain answers, considering sentence similarity and potential breakpoints between less related sentences.
  5. Fine-grained Filtering: This phase further refines the text by reducing noise and ambiguity. It addresses two issues:
  6. Abbreviation Expansion: Uses the Ab3P tool to expand local abbreviations, which can be crucial keywords but also ambiguous.
  7. Redundant Text Removal: Applies rules to cut off text following phrases like "such as," "for example," text in brackets, and non-enumerated lists after colons.
  8. BART-based Summary Generation: The filtered and combined text is then fed into the BART model, which generates the abstractive summary. BART's encoder processes the input bidirectionally, while its decoder generates the summary autoregressively. Beam search with a beamwidth of 5 is used for decoding.

Datasets

The primary dataset used for training and evaluation is the MEDIQA-AnS Dataset. This dataset was created for the MEDIQA 2021 shared task and includes original answers generated by the CHiQA medical question-answering system. Both extractive and abstractive summaries, manually created by medical experts, are available for this dataset.

Key Insights and Ideas

  • Question-Driven Filtering is Crucial: The paper strongly emphasizes that simply feeding raw text to a powerful model like BART is insufficient for question-answering summarization. Explicitly orienting the summarization process towards the question through filtering significantly improves relevance and coherence.
  • Coarse-grained Filtering is the Biggest Contributor: Ablation studies revealed that the coarse-grained filtering phase had the most substantial positive impact on the model's performance, highlighting the importance of selecting relevant information early in the pipeline.
  • Fine-grained Filtering Offers Marginal Gains: While the fine-grained filtering steps (abbreviation expansion and rule-based cut-offs) are conceptually sound, their contribution to the overall ROUGE-2 F1 score was found to be minimal. This suggests that the initial selection of relevant content is more critical than fine-tuning the exact wording.
  • Abstractive Summarization's Potential: The success of the SSG model, even with its limitations, demonstrates the growing capability of abstractive methods to generate more natural and coherent summaries compared to purely extractive approaches.

Key Results

The UETfishes team achieved significant success in the MEDIQA 2021 shared task:

  • Competition Performance: The SSG model ranked 2nd when evaluated on extractive references and 3rd when evaluated on abstractive references, demonstrating strong performance in a competitive setting.
  • ROUGE Scores: The best performance achieved was a ROUGE-2 F1 of 0.470 on extractive references and 0.147 on abstractive references.
  • Comparison with State-of-the-Art: The SSG model outperformed other strong abstractive summarization models like DistilBART and PEGASUS on this specific biomedical dataset, suggesting its tailored approach was effective.
  • Ablation Study: The study showed that all model components contributed to performance improvement, with coarse-grained filtering being the most impactful. Techniques like preventing 3-gram repetition, sampling, and beam search also played a role in boosting results.

Limitations and Future Work

The authors acknowledge several limitations and suggest avenues for future research:

  • Generalization of Input Content: A major challenge is the generalization of input content, especially for answer summarization systems. The model sometimes struggles to synthesize diverse information into a concise answer that directly addresses the question, as seen in an example where the model's summary was less precise than the ideal ground truth.
  • Ground Truth Dependency: The quality and style of the "golden data" (human-generated summaries) can lead to inconsistencies, as abstractors may use varied expressions, synonyms, and paraphrasing.
  • Sentence Selection and Cutting: The cosine similarity metric for sentence selection can be less effective with documents containing many sentences, potentially missing important content that doesn't have a high semantic similarity to the question. Fine-grained filtering strategies might also inadvertently remove important information.
  • Future Directions: The authors propose several directions for future work, including:
  • Applying machine learning models for more sophisticated sentence selection.
  • Deepening the question-analyzing capabilities.
  • Incorporating sentence clustering techniques to better group related information.
  • Exploring alternative methods for sentence selection and cutting strategies.

The authors also plan to release their source code to promote reproducibility and facilitate further research in this area.

Summary updated

BibTeX
@inproceedings{le2021uetfishes,
  author = {Hoang-Quynh Le and Quoc-An Nguyen and Quoc-Hung Duong and Minh-Quang Nguyen and Huy-Son Nguyen and Tam Doan Thanh and Hai-Yen Thi Vuong and Trang M. Nguyen},
  title = {{UETfishes at MEDIQA 2021: Standing-on-the-Shoulders-of-Giants Model for Abstractive Multi-answer Summarization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure