Overview of BioASQ 2024: The Twelfth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering

Anastasios Nentidis, Georgios Katsimpras, Anastasia Krithara, Salvador Lima-López, Eulàlia Farré-Maduell, Martin Krallinger, Natalia V. Loukachevitch, Vera Davydova, Elena Tutubalina, Georgios Paliouras

TL;DR

The 12th BioASQ challenge highlights a major shift in biomedical AI, where Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) have become the standard for question answering and entity recognition. This year’s competition introduced new benchmarks for multilingual cardiology clinical notes and nested entity recognition in Russian and English, revealing that while LLMs are powerful, domain-specific fine-tuning remains essential for high-precision medical tasks.

What Problem Does This Paper Address?

Biomedical information is growing at an exponential rate, making it nearly impossible for human experts to keep up with the latest literature and clinical data. The BioASQ challenge addresses the need for automated systems that can:

  1. Semantically Index and Retrieve: Find the exact "needle in the haystack" across millions of PubMed abstracts.
  2. Answer Complex Questions: Provide precise "factoid," "list," or "summary" answers to expert-level queries.
  3. Extract Clinical Entities: Identify diseases and drugs in messy, multilingual clinical notes, including "nested" entities (where one entity exists inside another, like "eye" within "eye movement disorder").

The core challenge is the high stakes and high complexity of medical jargon, which often causes general-purpose AI models to hallucinate or miss critical nuances.

Methods

The 2024 challenge saw a near-total dominance of Transformer-based architectures and Generative AI. Key approaches included:

  • RAG (Retrieval-Augmented Generation): Most teams used a two-stage pipeline. First, they used traditional lexical search (BM25) or neural bi-encoders to find relevant documents. Then, they fed those documents into an LLM (like GPT-4, Llama-3, or Mixtral) to generate the final answer.
  • Phase A+ (Direct QA): A new sub-task this year required systems to provide answers without being given the relevant snippets first, testing the "internalized" knowledge of these models.
  • Ensemble NER: For named entity recognition (NER), top teams combined multiple models (e.g., RoBERTa with multi-head CRF layers) and used data augmentation techniques like window sliding to handle long clinical documents.
  • Zero-Shot vs. Supervised: Researchers compared the performance of "out-of-the-box" LLMs against models specifically trained on medical data (like PubMedBERT or BINDER).

Datasets

BioASQ 2024 utilized a mix of established and brand-new benchmarks:

  • Task 12b: Over 5,000 biomedical questions annotated with ground-truth answers and snippets.
  • Task Synergy: A "continuous" dataset where experts provide incremental feedback on system responses for developing problems like COVID-19 and pediatric sepsis.
  • MultiCardioNER: Introduced the CardioCCC corpus, a new collection of 508 cardiology clinical cases in Spanish, English, and Italian.
  • BioNNE: A new dataset for nested NER based on the NEREL-BIO corpus, featuring parallel Russian and English PubMed abstracts.

Key Insights and Ideas

  • The "Cardiology Gap": In the MultiCardioNER task, systems that used cardiology-specific data (CardioCCC) significantly outperformed those trained on general medical data. This proves that even within medicine, "domain adaptation" to specific specialties is crucial.
  • Nested Entities are the Final Frontier for NER: While "flat" NER is largely solved, nested structures (BioNNE) remain difficult. Supervised models like BINDER (using XLM-RoBERTa) crushed zero-shot LLMs, which struggled to understand the hierarchical relationship between overlapping entities.
  • LLMs as Search Engines: Several teams used LLMs not just to answer questions, but to expand search queries, helping traditional search engines find more relevant documents by suggesting synonyms and related medical concepts.
  • The Italian Deficit: The paper noted a lack of specialized clinical models for Italian compared to Spanish and English, forcing teams to rely on cross-lingual transfer or machine translation.

Key Results

  • Question Answering: In Task 12b, top systems achieved a perfect 1.00 F1 score on "yes/no" questions in some batches, suggesting that binary medical QA is reaching a high level of maturity.
  • Factoid/List Questions: These remain harder. While performance is improving, the scores for "list" questions (where multiple items must be retrieved) showed more room for growth compared to "yes/no" queries.
  • MultiCardioNER: The best systems achieved an F1-score of ~0.82 for disease detection in Spanish and ~0.92 for drug detection.
  • BioNNE: The top F1-score for nested NER was 0.70, highlighting the significant difficulty of this new task compared to standard entity extraction.

Limitations and Future Work

The authors acknowledge that while "yes/no" performance is excellent, factoid and list-type questions still fluctuate in accuracy across different batches. There is also a clear need for more non-English clinical models, particularly for languages like Italian where specialized pre-trained models are scarce.

Future editions of BioASQ plan to expand into more diverse document types (beyond just abstracts) and even more focused sub-domains of medicine to continue pushing the boundaries of what biomedical AI can handle.

Summary updated

Abstract

This is an overview of the twelfth edition of the BioASQ challenge in the context of the Conference and Labs of the Evaluation Forum (CLEF) 2024. BioASQ is a series of international challenges promoting advances in large-scale biomedical semantic indexing and question answering. This year, BioASQ consisted of new editions of the two established tasks b and Synergy, and two new tasks: a) MultiCardioNER on the adaptation of clinical entity detection to the cardiology domain in a multilingual setting, and b) BIONNE on nested NER in Russian and English. In this edition of BioASQ, 37 competing teams participated with more than 700 distinct submissions in total for the four different shared tasks of the challenge. Similarly to previous editions, most of the participating systems achieved competitive performance, suggesting the continuous advancement of the state-of-the-art in the field.
BibTeX
@inproceedings{nentidis2024overview,
  author = {Anastasios Nentidis and Georgios Katsimpras and Anastasia Krithara and Salvador Lima-López and Eulàlia Farré-Maduell and Martin Krallinger and Natalia V. Loukachevitch and Vera Davydova and Elena Tutubalina and Georgios Paliouras},
  title = {{Overview of BioASQ 2024: The Twelfth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering}},
  booktitle = {CLEF},
  year = {2024},
  url = {https://openreview.net/forum?id=OA7H9Yjqt6},
}

Loading related content...

Figure