Introduction
While Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, their performance is often disproportionately high in English compared to lower-resource languages. For the 272 million Bengali speakers worldwide, this "digital divide" poses a significant barrier to AI-assisted education. To address this, researchers have introduced BEnQA, the first parallel Bengali-English benchmark specifically designed for science question answering.
Methodology: Bridging the Language Gap
The creation of BEnQA involved a multi-stage pipeline to ensure high-quality, aligned data from the Bangladeshi national curriculum (grades 8, 10, and 12).
The team digitized print-only examination papers and employed an automated alignment strategy. Bengali questions were translated via API, and their English counterparts were identified using cosine similarity of text embeddings (OpenAI’s text-embedding-ada-002). This process was finalized with manual verification by native speakers to ensure semantic equivalence.
The BEnQA Dataset
The resulting dataset comprises 5,161 multiple-choice questions spanning Physics, Chemistry, Biology, and Mathematics. A key feature of BEnQA is its categorization into three cognitive levels: Factual Knowledge, Procedural & Application, and Reasoning.
This categorization allows for a granular analysis of where LLMs succeed or fail. For instance, while models may recall facts easily, multi-step reasoning in a non-Latin script presents a much greater challenge than in English.
Key Insights and Performance Analysis
The study revealed several critical insights into the current state of multilingual AI:
- The Language Tax: A significant performance disparity exists between English and Bengali. GPT-3.5, for example, scored approximately 59% on 12th-grade Math in English but dropped to 43% in Bengali.
- The "Translation-Appended" Boost: One of the most significant findings is that appending an English translation to a Bengali prompt improves accuracy. This suggests that the models' "reasoning engine" is more effectively triggered by English text, even when the required output is in Bengali.
- Chain-of-Thought (CoT) Utility: CoT prompting significantly improved scores (10–20%) for reasoning and application questions but offered negligible benefits for factual recall.
- Open-Source Limitations: While proprietary models like GPT-4 and Claude 2.1 showed relative resilience, open-source models such as LLaMA-2 and Mistral struggled significantly with Bengali, highlighting a gap in the open-weights ecosystem for non-Latin scripts.
Conclusion and Future Work
BEnQA serves as a vital tool for measuring and closing the performance gap in Bengali AI. While GPT-4 currently leads the benchmark, the "language gap" remains a hurdle for most models. Future work will focus on incorporating visual reasoning (diagrams and figures) and improving instruction-following capabilities in low-resource languages to further democratize AI-assisted science education.
Summary updated