BEnQA: A Question Answering Benchmark for Bengali and English

Sheikh Shafayat, H. M. Quamran Hasan, Minhajur Rahman Chowdhury Mahim, Rifki Afina Putri, James Thorne, Alice OhACL 2024 · DOI

Introduction

While Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, their performance is often disproportionately high in English compared to lower-resource languages. For the 272 million Bengali speakers worldwide, this "digital divide" poses a significant barrier to AI-assisted education. To address this, researchers have introduced BEnQA, the first parallel Bengali-English benchmark specifically designed for science question answering.

Methodology: Bridging the Language Gap

The creation of BEnQA involved a multi-stage pipeline to ensure high-quality, aligned data from the Bangladeshi national curriculum (grades 8, 10, and 12).

Figure 1: The automated alignment pipeline used to create the parallel Bengali-English corpus.

The team digitized print-only examination papers and employed an automated alignment strategy. Bengali questions were translated via API, and their English counterparts were identified using cosine similarity of text embeddings (OpenAI’s text-embedding-ada-002). This process was finalized with manual verification by native speakers to ensure semantic equivalence.

The BEnQA Dataset

The resulting dataset comprises 5,161 multiple-choice questions spanning Physics, Chemistry, Biology, and Mathematics. A key feature of BEnQA is its categorization into three cognitive levels: Factual Knowledge, Procedural & Application, and Reasoning.

Figure 2: Distribution of questions by cognitive category and subject-wise breakdown across grades.

This categorization allows for a granular analysis of where LLMs succeed or fail. For instance, while models may recall facts easily, multi-step reasoning in a non-Latin script presents a much greater challenge than in English.

Key Insights and Performance Analysis

The study revealed several critical insights into the current state of multilingual AI:

  • The Language Tax: A significant performance disparity exists between English and Bengali. GPT-3.5, for example, scored approximately 59% on 12th-grade Math in English but dropped to 43% in Bengali.
  • The "Translation-Appended" Boost: One of the most significant findings is that appending an English translation to a Bengali prompt improves accuracy. This suggests that the models' "reasoning engine" is more effectively triggered by English text, even when the required output is in Bengali.
  • Chain-of-Thought (CoT) Utility: CoT prompting significantly improved scores (10–20%) for reasoning and application questions but offered negligible benefits for factual recall.
  • Open-Source Limitations: While proprietary models like GPT-4 and Claude 2.1 showed relative resilience, open-source models such as LLaMA-2 and Mistral struggled significantly with Bengali, highlighting a gap in the open-weights ecosystem for non-Latin scripts.

Conclusion and Future Work

BEnQA serves as a vital tool for measuring and closing the performance gap in Bengali AI. While GPT-4 currently leads the benchmark, the "language gap" remains a hurdle for most models. Future work will focus on incorporating visual reasoning (diagrams and figures) and improving instruction-following capabilities in low-resource languages to further democratize AI-assisted science education.

Summary updated

BibTeX
@inproceedings{shafayat2024benqa,
  author = {Sheikh Shafayat and H. M. Quamran Hasan and Minhajur Rahman Chowdhury Mahim and Rifki Afina Putri and James Thorne and Alice Oh},
  title = {{BEnQA: A Question Answering Benchmark for Bengali and English}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.68},
}
Figure