Bangla is the seventh most spoken language globally, with over 230 million speakers. Despite its vast user base, it remains classified as an "under-resourced" language in the field of Natural Language Processing (NLP). While English-centric AI benefits from a wealth of complex, high-quality datasets, Bangla researchers have historically relied on small, niche datasets or machine-translated versions of English benchmarks. These translated datasets often suffer from linguistic artifacts and fail to capture the cultural and grammatical nuances of native Bangla.
A new study introduces BanglaRQA, a large-scale, human-annotated dataset designed to move Bangla AI beyond simple fact-finding and into the realm of true reading comprehension. By incorporating diverse question types and adversarial unanswerable questions, BanglaRQA provides a robust benchmark for evaluating the state of Bangla language models.
The Problem: Beyond Translation and Factoids
Existing Bangla Question Answering (QA) datasets face three primary hurdles. First, translation artifacts from datasets like SQuAD often result in awkward phrasing that does not reflect natural Bangla usage. Second, there is a significant lack of diversity; most datasets focus exclusively on "factoid" questions (Who? What? When?) where the answer is a single, continuous string of text. Finally, many datasets ignore the "answerability" trap, assuming every question has an answer within the provided text. This encourages models to hallucinate answers when the necessary information is missing.
As shown in the comparison above, BanglaRQA is the first dataset to combine human curation with support for unanswerable questions, list questions, confirmation (Yes/No) questions, and multiple span answers, filling a critical gap in the research landscape.
Methodology: Building a Native Benchmark
The researchers developed BanglaRQA through a rigorous six-step pipeline to ensure high quality and linguistic authenticity:
- Human Curation: Undergraduate engineering students manually collected 3,000 passages from Bangla Wikipedia across 20 diverse domains, including politics, sports, and science.
- Diverse Question Types: Annotators were tasked with creating four distinct question categories: Factoid (specific answers), Causal ("Why" or "How" questions), Confirmation (Yes/No), and List (multiple entities).
- Adversarial Unanswerability: To prevent model hallucination, annotators created questions that appear relevant to the passage but cannot be answered by it. This forces models to learn the critical "I don't know" response.
- Modeling and Tagging: The dataset was benchmarked using four state-of-the-art Transformer models: BanglaT5, mT5, BanglaBERT, and mBERT. For extractive models, a B-I-O (Beginning, Inside, Outside) tagging approach was employed to handle complex answers spanning multiple, non-contiguous parts of the text.
Key Results and Insights
The benchmarking process established a strong baseline for future Bangla NLP research. BanglaT5 emerged as the top-performing model, achieving an overall Exact Match (EM) of 62.42% and an F1 score of 78.11%.
The study revealed several key insights into model behavior:
- Question Difficulty: Confirmation questions were the easiest for models to handle (86.8% F1), while List questions (65.2% F1) and Multiple Span answers (55.7% F1) proved the most challenging, indicating areas for future architectural improvement.
- Robustness to Unanswerability: The models achieved an impressive 85.36% F1 score in identifying unanswerable questions, demonstrating that the adversarial training effectively reduced the tendency to guess.
- Superior Generalization: A significant finding was that models trained on BanglaRQA outperformed those trained on the translated
bn_squaddataset when tested onbn_squaditself. This suggests that the human-curated diversity of BanglaRQA creates more robust and adaptable models than translated data alone.
Conclusion and Future Work
BanglaRQA represents a major milestone for Bangla NLP, providing a high-quality, 15,000-pair dataset that reflects the complexity of real-world reading comprehension. While the dataset is smaller than English counterparts like SQuAD, its focus on human nuance and diverse answer types provides a superior foundation for regional AI development.
Future research will focus on expanding the dataset's scale and exploring automated "Question Generation" techniques to augment training data without the high cost of manual human annotation. By bridging the resource gap, BanglaRQA paves the way for more sophisticated and culturally aware AI tools for the global Bangla-speaking community.
Summary updated