Overview of BioASQ 2024: The twelfth BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering

Anastasios Nentidis, Georgios Katsimpras, Anastasia Krithara, Salvador Lima-López, Eulàlia Farré-Maduell, Martin Krallinger, Natalia V. Loukachevitch, Vera Davydova, Elena Tutubalina, Georgios Paliouras

Abstract

This is an overview of the twelfth edition of the BioASQ challenge in the context of the Conference and Labs of the Evaluation Forum (CLEF) 2024. BioASQ is a series of international challenges promoting advances in large-scale biomedical semantic indexing and question answering. This year, BioASQ consisted of new editions of the two established tasks b and Synergy, and two new tasks: a) MultiCardioNER on the adaptation of clinical entity detection to the cardiology domain in a multilingual setting, and b) BIONNE on nested NER in Russian and English. In this edition of BioASQ, 37 competing teams participated with more than 700 distinct submissions in total for the four different shared tasks of the challenge. Similarly to previous editions, most of the participating systems achieved competitive performance, suggesting the continuous advancement of the state-of-the-art in the field.

Introduction

The exponential growth of biomedical information presents a significant challenge for human experts attempting to stay current with literature and clinical data. The BioASQ challenge addresses this by fostering the development of automated systems for semantic indexing, precise question answering (factoid, list, and summary), and clinical entity extraction. The 2024 edition emphasizes the transition toward Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) to handle the complexity of medical jargon and the high stakes of clinical accuracy.

Methodology and Participating Systems

The 2024 challenge observed a dominant shift toward Transformer-based architectures and Generative AI. Most participating teams implemented two-stage pipelines utilizing RAG. This typically involved an initial retrieval phase using lexical search (BM25) or neural bi-encoders, followed by a generation phase using state-of-the-art LLMs such as GPT-4, Llama-3, or Mixtral.

Overview of participating systems and their technical approaches, highlighting the prevalence of LLMs and RAG.

A new sub-task, Phase A+, tested the "internalized" knowledge of models by requiring answers without provided snippets. For Named Entity Recognition (NER), ensembles combining models like RoBERTa with multi-head CRF layers were common, alongside data augmentation techniques to process long clinical documents.

Datasets and Tasks

BioASQ 2024 utilized several benchmarks to push the boundaries of biomedical AI:

  • Task 12b: Over 5,000 biomedical questions with ground-truth answers.
  • Task Synergy: A continuous dataset with expert feedback on evolving medical problems.
  • MultiCardioNER: Focused on the cardiology domain across Spanish, English, and Italian.
  • BioNNE: Addressed nested NER using the NEREL-BIO corpus in Russian and English.
Statistics of the MultiCardioNER datasets, including the new CardioCCC corpus across multiple languages.

Key Insights and Results

The competition revealed several critical findings regarding the current state of the field:

  • Domain Adaptation: In the MultiCardioNER task, systems utilizing cardiology-specific data (CardioCCC) outperformed general medical models, highlighting the necessity of specialty-specific fine-tuning.
  • Nested NER Challenges: While binary QA is maturing (with some systems achieving perfect F1 scores in Task 12b), nested entity recognition remains difficult. Supervised models like BINDER significantly outperformed zero-shot LLMs in identifying hierarchical relationships.
  • LLMs in Retrieval: Beyond generation, LLMs were effectively used for query expansion, aiding traditional search engines in finding relevant documents through medical concept suggestion.
  • Performance Metrics: Top systems achieved F1-scores of ~0.82 for disease detection and ~0.92 for drug detection in Spanish. However, the BioNNE task's top F1-score of 0.70 underscores the complexity of nested structures.

Conclusion

While performance in binary question answering is reaching maturity, factoid and list-type questions continue to show variability. There remains a significant need for specialized clinical models in languages other than English and Spanish, such as Italian. Future BioASQ editions aim to incorporate more diverse document types and deeper sub-domain focus to continue pushing the boundaries of what biomedical AI can handle.

Summary updated

Abstract

This is an overview of the twelfth edition of the BioASQ challenge in the context of the Conference and Labs of the Evaluation Forum (CLEF) 2024. BioASQ is a series of international challenges promoting advances in large-scale biomedical semantic indexing and question answering. This year, BioASQ consisted of new editions of the two established tasks b and Synergy, and two new tasks: a) MultiCardioNER on the adaptation of clinical entity detection to the cardiology domain in a multilingual setting, and b) BIONNE on nested NER in Russian and English. In this edition of BioASQ, 37 competing teams participated with more than 700 distinct submissions in total for the four different shared tasks of the challenge. Similarly to previous editions, most of the participating systems achieved competitive performance, suggesting the continuous advancement of the state-of-the-art in the field.
BibTeX
@inproceedings{nentidis2024overview,
  author = {Anastasios Nentidis and Georgios Katsimpras and Anastasia Krithara and Salvador Lima-López and Eulàlia Farré-Maduell and Martin Krallinger and Natalia V. Loukachevitch and Vera Davydova and Elena Tutubalina and Georgios Paliouras},
  title = {{Overview of BioASQ 2024: The twelfth BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering}},
  booktitle = {CLEF},
  year = {2024},
  url = {https://openreview.net/forum?id=OA7H9Yjqt6},
}
Figure