NCUEE-NLP at BioLaySumm Task 2: Readability-Controlled Summarization of Biomedical Articles Using the PRIMERA Models

Chao-Yi Chen, Jen-Hao Yang, Lung-Hao LeeACL 2023

TL;DR

This paper presents the NCUEE-NLP system for the BioLaySumm Task 2, which aims to generate both technical abstracts and lay summaries from biomedical articles. By fine-tuning the PRIMERA model, they achieved top rankings in relevance and readability, demonstrating the model's effectiveness in making complex scientific information more accessible.

What Problem Does This Paper Address?

Biomedical articles are dense with technical jargon, making them difficult for non-experts to understand. Automatic text summarization, particularly abstractive summarization, offers a solution by generating concise summaries that convey essential information. However, creating summaries that are both technically accurate and easily readable for a general audience is a significant challenge. The BioLaySumm Task 2 specifically addresses this by requiring systems to generate two types of summaries: a technical abstract for experts and a "lay summary" for a broader audience, focusing on readability.

Methods

The NCUEE-NLP system leverages the PRIMERA (PYRAMID-based Masked Sentence Pre-training for Multi-document Summarization) model. PRIMERA is a pre-trained transformer model designed to reduce the need for extensive task-specific architectures and labeled data. It utilizes a novel pre-training strategy called Entity Pyramid, which identifies and aggregates salient information based on entity frequency across document clusters, prioritizing sentences representative of more documents. The model is initialized with the Longformer-Encoder-Decoder (LED) architecture to handle long input sequences.

For BioLaySumm Task 2, the researchers fine-tuned the PRIMERA model separately for generating technical abstracts and lay summaries. They treated each section of a biomedical article (excluding acknowledgments and references) as a distinct document, delimited by a <doc-sep> token, to accommodate the model's multi-document summarization capabilities. During evaluation, they generated both summary types independently and then combined them for the final submission.

Datasets

The study utilized datasets provided by the BioLaySumm task organizers. This included a collection of biomedical research articles from the Public Library of Science (PLOS), along with expert-written technical abstracts and lay summaries.

  • Training Data: 24,773 training articles were used to fine-tune the pre-trained language models.
  • Validation Data: 1,376 articles were used for system development and parameter optimization.
  • Test Data: 142 articles were used for final system evaluation.

The average token counts for technical abstracts and lay summaries in the training set were 268.11 and 194.90, respectively.

Key Insights and Ideas

  • PRIMERA's Adaptability: The PRIMERA model, originally designed for multi-document summarization, proved to be highly effective when adapted for single-document summarization tasks like generating technical abstracts and lay summaries.
  • Separate Fine-tuning for Summary Types: Fine-tuning PRIMERA independently for technical abstracts and lay summaries allowed the model to specialize in the distinct requirements of each output, leading to better performance.
  • Lay Summarization is More Challenging: The analysis revealed that summarization performance for technical abstracts generally surpassed that of lay summaries, highlighting the increased complexity of generating readable content for a general audience.
  • Factuality Concerns: While PRIMERA excelled in relevance and readability, it underperformed in factuality. This suggests that ensuring factual accuracy in generated summaries, especially for complex scientific content, remains a significant challenge.

Key Results

The NCUEE-NLP system, powered by the fine-tuned PRIMERA model, achieved strong performance in the BioLaySumm Task 2:

  • Overall Performance: The system tied for first place in overall performance.
  • Relevance: Ranked first among all submissions.
  • Readability: Ranked second.
  • Factuality: Ranked fourth.

The paper also compared PRIMERA with other models like PEGASUS and BART-Longformer. On the test set, PRIMERA outperformed the others in relevance and readability, though BART-Longformer showed a slight edge over PEGASUS.

Limitations and Future Work

The authors acknowledge several limitations:

  • Novelty of Techniques: This study represents an initial exploration of the research topic. The primary limitation is the novelty of the techniques employed, as they relied on existing pre-trained models (PRIMERA, PEGASUS, BART-Longformer) downloaded from Hugging Face.
  • Inappropriate Pre-training: The pre-trained models might not be ideally suited for this specific summarization task.
  • Limited Data Augmentation: The fine-tuning was performed solely on the task-given datasets, without incorporating additional related summarization data to potentially enhance model performance.

Future work could involve exploring more advanced pre-training strategies or investigating methods to improve the factuality of generated summaries, especially given the observed underperformance in this area. The authors also plan to conduct error analysis once the evaluation code and gold standard for the test set are publicly available to better understand the reasons behind the factuality issues.

Summary updated

BibTeX
@inproceedings{chen2023ncuee,
  author = {Chao-Yi Chen and Jen-Hao Yang and Lung-Hao Lee},
  title = {{NCUEE-NLP at BioLaySumm Task 2: Readability-Controlled Summarization of Biomedical Articles Using the PRIMERA Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure