ISIKSumm at BioLaySumm Task 1: BART-based Summarization System Enhanced with Bio-Entity Labels

Cagla Colak, Lknur KaradenizACL 2023

TL;DR

This paper presents ISIKSumm, a system for generating lay summaries of biomedical articles by fine-tuning the BART model. By incorporating bio-entity labeling using the Stanza library, ISIKSumm aims to improve the accessibility of scientific research for a general audience by simplifying technical jargon.

What Problem Does This Paper Address?

Communicating complex scientific research to the general public is a significant challenge. Biomedical articles, in particular, are dense with specialized terminology (like gene and protein names) that can be a barrier to understanding for non-experts. The BioLaySumm task aims to automate the creation of "lay summaries" – simplified versions of research findings – to bridge this gap. This paper addresses the challenge of making these summaries accessible by focusing on the accurate identification and simplification of technical terms within biomedical texts.

Methods

The core of the ISIKSumm system is the BART (Bidirectional and Auto-Regressive Transformer) model, a powerful sequence-to-sequence model for text generation. The authors employ two main approaches:

  • Model 1: This model preprocesses the input article by cleaning text and then uses the Stanza library's named entity recognition (NER) capabilities to identify and tag bio-entities (e.g., proteins, genes, cell types). These identified entities are then replaced with simpler explanations before being fed into the BART model for summarization. This approach aims to inject domain-specific knowledge into the summarization process.
  • Model 2: This approach uses the original abstract as a summary and combines it with a summary generated by a BART-based model trained on the pre-processed full text. Due to BART's input length limitations, articles are divided into sections, summarized individually, and then concatenated. Post-processing includes text cleaning and bio-entity tagging.

Datasets

The BioLaySumm task provides two datasets curated from two biomedical journals: PLOS and eLife. These datasets consist of biomedical articles and their corresponding lay summaries, used for training and evaluating the summarization models.

Key Insights and Ideas

  • Bio-entity labeling is crucial for lay summaries: The paper strongly suggests that explicitly identifying and handling technical terms (bio-entities) is key to improving the understandability of biomedical summaries for a non-expert audience.
  • Stanza's biomedical NER is effective: The authors found that Stanza's specialized biomedical NER models (BC5CDR and JNLPBA) performed well in identifying technical terms, outperforming other models in previous studies.
  • Pre-processing enhances summarization quality: Both models, particularly Model 1, show improved performance when incorporating bio-entity labeling as a pre-processing step, indicating that simplifying technical jargon before summarization is beneficial.

Key Results

The paper reports promising results on the BioLaySumm task. While specific comparative numbers between Model 1 and Model 2 are presented, the general trend indicates that:

  • Model 1 (with bio-entity enhancement) generally outperforms a baseline BART model without such enhancements. For instance, in Table 4, Model 1+Enh shows higher precision, recall, and F1 scores compared to Model 1-Enh.
  • Model 2 also shows competitive performance, sometimes outperforming Model 1. Table 5 indicates that Model 2 achieved slightly better ROUGE scores on the eLife dataset compared to Model 1+Enh, although Model 1+Enh had better BERTScore.
  • The authors achieved competitive results on both PLOS and eLife datasets, with average ROUGE-1, ROUGE-2, and ROUGE-L scores of 0.379, 0.070, and 0.353 respectively.

Limitations and Future Work

Limitations:

  • BART's input length constraint: The maximum sequence length of 1024 characters for the BART model required dividing articles into sections, which could potentially lead to less coherent summaries.
  • Computational resources: Experiments were conducted on Google Colab, which has time and memory limitations, and occasional connection issues could slow down progress.

Future Work:

  • Exploring additional pre/post-processing techniques.
  • Investigating alternative approaches for fine-tuning the BART model.
  • Developing methods to address the challenges of generating accurate and accessible lay summaries for highly complex scientific articles, particularly concerning the translation of intricate terminology without losing critical details.

Summary updated

BibTeX
@inproceedings{colak2023isiksumm,
  author = {Cagla Colak and Lknur Karadeniz},
  title = {{ISIKSumm at BioLaySumm Task 1: BART-based Summarization System Enhanced with Bio-Entity Labels}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure