BanglaRQA: A Benchmark Dataset for Under-resourced Bangla Language Reading Comprehension-based Question Answering with Diverse Question-Answer Types

Syed Mohammed Sartaj Ekram, Adham Arik Rahman, Md. Sajid Altaf, Mohammed Saidul Islam, Mehrab Mustafy Rahman, Md Mezbaur Rahman, Md Azam Hossain, Abu Raihan Mostofa KamalEMNLP 2022 · DOI

Bangla is the seventh most spoken language globally, with over 230 million speakers. Despite its vast user base, it remains classified as an "under-resourced" language in the field of Natural Language Processing (NLP). While English-centric AI benefits from a wealth of complex, high-quality datasets, Bangla researchers have historically relied on small, niche datasets or machine-translated versions of English benchmarks. These translated datasets often suffer from linguistic artifacts and fail to capture the cultural and grammatical nuances of native Bangla.

A new study introduces BanglaRQA, a large-scale, human-annotated dataset designed to move Bangla AI beyond simple fact-finding and into the realm of true reading comprehension. By incorporating diverse question types and adversarial unanswerable questions, BanglaRQA provides a robust benchmark for evaluating the state of Bangla language models.

The Problem: Beyond Translation and Factoids

Existing Bangla Question Answering (QA) datasets face three primary hurdles. First, translation artifacts from datasets like SQuAD often result in awkward phrasing that does not reflect natural Bangla usage. Second, there is a significant lack of diversity; most datasets focus exclusively on "factoid" questions (Who? What? When?) where the answer is a single, continuous string of text. Finally, many datasets ignore the "answerability" trap, assuming every question has an answer within the provided text. This encourages models to hallucinate answers when the necessary information is missing.

Table 1: Comparison of BanglaRQA with existing Bangla QA datasets, highlighting its unique support for diverse question types and unanswerable questions.

As shown in the comparison above, BanglaRQA is the first dataset to combine human curation with support for unanswerable questions, list questions, confirmation (Yes/No) questions, and multiple span answers, filling a critical gap in the research landscape.

Methodology: Building a Native Benchmark

The researchers developed BanglaRQA through a rigorous six-step pipeline to ensure high quality and linguistic authenticity:

  1. Human Curation: Undergraduate engineering students manually collected 3,000 passages from Bangla Wikipedia across 20 diverse domains, including politics, sports, and science.
  2. Diverse Question Types: Annotators were tasked with creating four distinct question categories: Factoid (specific answers), Causal ("Why" or "How" questions), Confirmation (Yes/No), and List (multiple entities).
  3. Adversarial Unanswerability: To prevent model hallucination, annotators created questions that appear relevant to the passage but cannot be answered by it. This forces models to learn the critical "I don't know" response.
  4. Modeling and Tagging: The dataset was benchmarked using four state-of-the-art Transformer models: BanglaT5, mT5, BanglaBERT, and mBERT. For extractive models, a B-I-O (Beginning, Inside, Outside) tagging approach was employed to handle complex answers spanning multiple, non-contiguous parts of the text.

Key Results and Insights

The benchmarking process established a strong baseline for future Bangla NLP research. BanglaT5 emerged as the top-performing model, achieving an overall Exact Match (EM) of 62.42% and an F1 score of 78.11%.

Figure 1: Performance metrics (EM and F1) of the BanglaT5 model over 15 training epochs, showing steady improvement and stabilization.

The study revealed several key insights into model behavior:

  • Question Difficulty: Confirmation questions were the easiest for models to handle (86.8% F1), while List questions (65.2% F1) and Multiple Span answers (55.7% F1) proved the most challenging, indicating areas for future architectural improvement.
  • Robustness to Unanswerability: The models achieved an impressive 85.36% F1 score in identifying unanswerable questions, demonstrating that the adversarial training effectively reduced the tendency to guess.
  • Superior Generalization: A significant finding was that models trained on BanglaRQA outperformed those trained on the translated bn_squad dataset when tested on bn_squad itself. This suggests that the human-curated diversity of BanglaRQA creates more robust and adaptable models than translated data alone.

Conclusion and Future Work

BanglaRQA represents a major milestone for Bangla NLP, providing a high-quality, 15,000-pair dataset that reflects the complexity of real-world reading comprehension. While the dataset is smaller than English counterparts like SQuAD, its focus on human nuance and diverse answer types provides a superior foundation for regional AI development.

Future research will focus on expanding the dataset's scale and exploring automated "Question Generation" techniques to augment training data without the high cost of manual human annotation. By bridging the resource gap, BanglaRQA paves the way for more sophisticated and culturally aware AI tools for the global Bangla-speaking community.

Summary updated

BibTeX
@inproceedings{ekram2022banglarqa,
  author = {Syed Mohammed Sartaj Ekram and Adham Arik Rahman and Md. Sajid Altaf and Mohammed Saidul Islam and Mehrab Mustafy Rahman and Md Mezbaur Rahman and Md Azam Hossain and Abu Raihan Mostofa Kamal},
  title = {{BanglaRQA: A Benchmark Dataset for Under-resourced Bangla Language Reading Comprehension-based Question Answering with Diverse Question-Answer Types}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.186},
}
Figure