AUTH at BioLaySumm 2024: Bringing Scientific Content to Kids

Loukritia Stefanou, Tatiana Passali, Grigorios TsoumakasACL 2024

TL;DR

This paper tackles the challenge of making complex biomedical research accessible to children by generating lay summaries. They fine-tuned the BioBART-v2 model, incorporating special tokens and data augmentation with GPT-4, achieving a 7th place ranking in the BioLaySumm 2024 shared task.

What Problem Does This Paper Address?

The vast majority of scientific research, particularly in fields like biomedicine, is written in highly technical language that is inaccessible to the general public, including children. This creates a significant barrier to understanding scientific advancements, potentially leading to misinformation and a lack of public engagement with science. The BioLaySumm 2024 shared task specifically aims to bridge this gap by generating lay summaries of biomedical research articles suitable for a broad audience, with a particular emphasis on making them understandable to children.

Methods

The core of the approach is the fine-tuning of BioBART-v2, a language model pre-trained on biomedical text. To enhance its performance for lay summarization, the researchers employed two key strategies:

  • Special Tokens: They introduced special tokens (<elife>, <plos>, <general_lay_summary>, <kids_lay_summary>) during training. These tokens help the model differentiate between the two source datasets (eLife and PLOS) and also signal whether the target summary should be a general lay summary or a simplified version for children. This allows for controllable generation.
  • Data Augmentation: To improve the readability and simplicity of summaries, especially for younger audiences, they used GPT-4 to rewrite complex lay summaries from the eLife and PLOS datasets. This augmentation process was guided by examples from "Science Journal for Kids" (SJK) to ensure the generated summaries were age-appropriate.

Datasets

The study utilized two primary datasets for the BioLaySumm 2024 shared task:

  • eLife Dataset: Contains articles from the eLife biomedical journal. The lay summaries in this dataset are often produced collaboratively by expert editors and authors, resulting in longer, more abstractive, and generally more readable summaries.
  • PLOS Dataset: Comprises articles from various Public Library of Science (PLOS) journals. The lay summaries here are written by the authors themselves and are typically shorter and more concise.

Additionally, the researchers created a new dataset by:

  • Science Journal for Kids (SJK) Dataset: This dataset was constructed by collecting kid-friendly adaptations of scientific papers from the SJK website. These articles use simpler vocabulary and relatable examples. The SJK dataset was used to guide the GPT-4 augmentation process for creating child-friendly summaries.

The final dataset for the shared task consisted of 300 pairs of biomedical abstracts and their corresponding lay summaries.

Key Insights and Ideas

  • Controllable Generation is Key: The use of special tokens proved crucial for guiding the model to generate summaries tailored to specific datasets (eLife vs. PLOS) and target audiences (general vs. kids). This fine-grained control is essential for adapting to the nuances of different summarization tasks.
  • Augmentation Boosts Readability, But with Caveats: While augmenting existing lay summaries with GPT-4 significantly improved readability scores (e.g., lowering Flesch-Kincaid Grade Level), it sometimes led to a slight decrease in relevance and factuality, especially for the PLOS dataset. This highlights a trade-off between simplicity and strict adherence to original content.
  • Dataset Characteristics Matter: The eLife dataset, with its more abstractive and editor-written summaries, seemed to provide richer learning material for the model, leading to more significant improvements in relevance during training compared to the PLOS dataset.
  • Bridging the Science-to-Public Gap is Crucial: The success of this approach, evidenced by their 7th place ranking, underscores the growing importance and feasibility of using advanced NLP models to make complex scientific information accessible to a wider audience, including children.

Key Results

The proposed approach achieved a 7th place ranking out of 57 participants in the BioLaySumm 2024 shared task.

  • Readability Improvements: The augmented summaries showed significant improvements in readability metrics. For example, the Flesch-Kincaid Grade Level for eLife summaries dropped from an average of 10.74 to 7.90, and for PLOS summaries from 14.73 to 8.57.
  • Relevance: The use of special tokens consistently improved relevance scores across both datasets.
  • Factuality: Factuality metrics showed a more complex pattern. While some improvement was observed with training, the introduction of augmented data sometimes led to a decline, particularly in later training steps.

Limitations and Future Work

The authors acknowledge the following limitations:

  • Factual Accuracy: Explicitly analyzing the factual accuracy of the generated summaries was not performed. This is a critical area for future work, especially in the biomedical domain.
  • Augmentation Trade-offs: The data augmentation process, while improving readability, sometimes compromised content relevance and factual accuracy.

Future research directions suggested include:

  • Integrating Factuality Metrics: Developing methods to evaluate and improve the factual accuracy of generated summaries.
  • Post-editing and Human Review: Incorporating post-editing techniques or human review to further refine summaries and remove inaccuracies.
  • Exploring Different Augmentation Strategies: Investigating alternative augmentation techniques that might better balance readability and factual accuracy.

Summary updated

BibTeX
@inproceedings{stefanou2024auth,
  author = {Loukritia Stefanou and Tatiana Passali and Grigorios Tsoumakas},
  title = {{AUTH at BioLaySumm 2024: Bringing Scientific Content to Kids}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure