BanglaMedQA and BanglaMMedBench: Evaluating Retrieval-Augmented Generation Strategies for Bangla Biomedical Question Answering

Sadia Sultana, Saiyma Sittul Muna, Mosammat Zannatul Samarukh, Ajwad Abrar, Tareque Mohmud ChowdhuryarXiv 2025 · arXiv · OpenReview

TL;DR

This paper introduces two new large-scale datasets, BanglaMedQA and BanglaMMedBench, for evaluating biomedical question answering in Bangla, a low-resource language. They benchmark various Retrieval-Augmented Generation (RAG) strategies, finding that an "Agentic RAG" approach, which dynamically chooses retrieval methods, achieves the highest accuracy and best rationale quality.

What Problem Does This Paper Address?

Access to reliable medical knowledge is crucial, but current AI systems often struggle with low-resource languages like Bangla. This limits equitable access to healthcare information for the over 230 million Bangla speakers worldwide. Specifically, building accurate biomedical Question Answering (QA) systems for Bangla faces challenges due to a lack of high-quality, domain-specific datasets and the difficulty of adapting existing advanced NLP techniques. Existing RAG methods, while effective in English, haven't been thoroughly explored or adapted for the nuances of Bangla medical QA, especially for multiple-choice question formats that require reasoning and rationale generation.

Methods

The core of this work involves developing and evaluating several Retrieval-Augmented Generation (RAG) strategies. RAG combines information retrieval with text generation to improve factual accuracy. The authors implemented and compared:

  • Traditional RAG: Retrieves top-k passages from a knowledge base.
  • RAG with Zero-Shot Fallback: If retrieval fails, it falls back to a zero-shot generation approach.
  • Web Search with Zero-Shot Fallback: Uses web search for retrieval, with zero-shot fallback.
  • Agentic RAG: A more sophisticated approach where a router dynamically decides whether to use local retrieval, web retrieval, or zero-shot reasoning based on context sufficiency.
  • Iterative Feedback RAG: Refines queries and context through multiple feedback loops to improve accuracy.
  • Aggregate k-values RAG: Combines predictions from multiple retrieval depths (different k values) to enhance robustness.

A key novelty is the creation of a Bangla medical textbook corpus using Optical Character Recognition (OCR) to serve as a local retrieval source.

Datasets

The paper introduces two novel datasets:

  • BanglaMedQA: The first curated Bangla medical MCQ dataset, containing 1,000 questions from authentic Bangladeshi medical admission tests (MBBS, BDS, AFMC) from 1990-2024. Each question includes a correct answer and an explanation.
  • BanglaMMedBench: This dataset extends an English benchmark into Bangla by translating and refining approximately 1,000 situational and complex questions inspired by the USMLE. These questions are designed to test reasoning and come with rationales.

Key Insights and Ideas

  • Low-resource languages are a significant barrier: The lack of specialized datasets and adapted models for languages like Bangla hinders equitable access to AI-driven medical knowledge.
  • Agentic RAG is a promising direction: Dynamically selecting the best retrieval and reasoning strategy based on context is more effective than fixed approaches. This "smart" routing can adapt to different question types and information availability.
  • OCR is crucial for building domain-specific corpora: Digitizing physical resources like textbooks via OCR is essential for creating knowledge bases in low-resource languages.
  • Web search isn't always a silver bullet: While web search can augment knowledge, it doesn't always align with specialized, scenario-based medical questions, sometimes even reducing accuracy.
  • Model scale matters, but with diminishing returns: Larger LLMs generally perform better, but the gains can plateau, especially when moving beyond certain model sizes.

Key Results

  • Agentic RAG achieved the highest accuracy: 89.54% using the openai/gpt-oss-120b model.
  • Rationale quality is also improved by Agentic RAG: It showed superior ROUGE-1, ROUGE-L, and BLEU scores, indicating strong linguistic and semantic coherence.
  • Zero-Shot serves as a baseline: Accuracies ranged from 42.15% to 82.60% depending on the model.
  • Traditional RAG and RAG with Zero-Shot Fallback showed significant improvements over Zero-Shot, reaching up to 86.32% and 87.22% respectively.
  • LLaMA models benefited more from retrieval-based augmentation compared to GPT-oss models, which already performed well in zero-shot settings.
  • Web Search sometimes decreased accuracy on the BanglaMMedBench dataset, particularly for scenario-based questions.

Limitations and Future Work

  • Dataset Size and Translation Quality: The authors acknowledge limitations in dataset size, translation quality, and computational resources.
  • Model Capacity and Initial Retrieval Quality: Iterative Feedback RAG's performance was sensitive to model capacity and the quality of the initial retrieval.
  • Web Search Limitations: Generic web content may not always align with specialized medical questions, potentially reducing accuracy.
  • Future Work: The paper establishes a foundation for future research in multilingual medical AI, suggesting further exploration of RAG strategies and their application to Bangla biomedical QA. The created datasets are intended to facilitate this ongoing research.

Summary updated

Abstract

Developing accurate biomedical Question Answering (QA) systems in low-resource languages remains a major challenge, limiting equitable access to reliable medical knowledge. This paper introduces BanglaMedQA and BanglaMMedBench, the first large-scale Bangla biomedical Multiple Choice Question (MCQ) datasets designed to evaluate reasoning and retrieval in medical artificial intelligence (AI). The study applies and benchmarks several Retrieval-Augmented Generation (RAG) strategies, including Traditional, Zero-Shot Fallback, Agentic, Iterative Feedback, and Aggregate RAG, combining textbook-based and web retrieval with generative reasoning to improve factual accuracy. A key novelty lies in integrating a Bangla medical textbook corpus through Optical Character Recognition (OCR) and implementing an Agentic RAG pipeline that dynamically selects between retrieval and reasoning strategies. Experimental results show that the Agentic RAG achieved the highest accuracy 89.54% with openai/gpt-oss-120b, outperforming other configurations and demonstrating superior rationale quality. These findings highlight the potential of RAG-based methods to enhance the reliability and accessibility of Bangla medical QA, establishing a foundation for future research in multilingual medical artificial intelligence.
BibTeX
@misc{sultana2025banglamedqa,
  author = {Sadia Sultana and Saiyma Sittul Muna and Mosammat Zannatul Samarukh and Ajwad Abrar and Tareque Mohmud Chowdhury},
  title = {{BanglaMedQA and BanglaMMedBench: Evaluating Retrieval-Augmented Generation Strategies for Bangla Biomedical Question Answering}},
  year = {2025},
  eprint = {2511.04560},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.04560},
}
Figure