BDBBQ: A Human-AI Collaborative Benchmark for Bengali Social Bias in LLMs

Provat Saha, Ashika Habib Khan, Tahseen Tanha, Hasin Ishrak, Muhammad Iqbal HossainarXiv 2026 · arXiv

TL;DR

BDBBQ is a new bilingual benchmark (English and Bengali) designed to evaluate social bias in Large Language Models (LLMs) within the specific sociocultural context of Bangladesh. The study reveals that cross-lingual accuracy and social bias are distinct dimensions; a model’s high performance in one language does not guarantee fairness in another, nor does high accuracy necessarily correlate with lower bias.

What Problem Does This Paper Address?

Most LLM bias benchmarks are English-centric and fail to capture the nuances of different cultures. Simple translation of these benchmarks into other languages, like Bengali, often misses the "surface meaning" and fails to account for local stereotypes, social institutions, and real-life scenarios unique to a region.

For example, a stereotype about a specific profession or regional affiliation in Bangladesh might not exist in a Western context. The authors address this gap by creating BDBBQ, a culturally grounded extension of the BBQ (Bias Benchmark for QA) framework specifically for the Bengali-speaking community in Bangladesh.

Methods

The researchers followed the BBQ framework, which uses a question-answering format to probe for bias. Their methodology includes:

  • Ambiguous vs. Disambiguated Contexts: Models are given short passages. In ambiguous contexts, the answer is unstated, and the correct response is "Unknown." If a model chooses a specific social group, it reveals an internal bias. In disambiguated contexts, the answer is explicitly provided to test if the model can override its biases when evidence is present.
  • Controlled Variation: To ensure robustness, the authors used balanced answer permutations, varied question framings (negative vs. non-negative), and multiple wording variants.
  • Human-AI Collaboration: The dataset was validated through structural checks, 220 statistical tests, and expert human annotation of 150 instances to ensure high inter-annotator agreement (Fleiss’ $\kappa = 0.8349$).
  • Scoring: They utilized two primary metrics: $s_{DIS}$ (directional bias in disambiguated contexts) and $s_{AMB}$ (bias in ambiguous contexts, scaled by the error rate).

Datasets

The primary contribution is the BDBBQ dataset, an expansion of the earlier, smaller BanglaBBQ.

  • Scale: 61,312 QA instances (30,656 each for English and Bengali).
  • Categories: It covers 12 social categories relevant to the Bangladeshi context, including Religion, Regional affiliation, Politics, and Occupational background.
Table I: Inventory of the BDBBQ dataset showing categories and record counts.

Key Insights and Ideas

  • The "Unknown" Gap: Models struggle significantly more in Bengali when the correct answer is "Unknown." For instance, Qwen 2.5 3B had 66.83% accuracy on ambiguous English questions but plummeted to 18.26% in Bengali.
  • Accuracy $\neq$ Fairness: A critical finding is that higher accuracy does not necessarily mean lower bias. A model might be very good at extracting facts but still exhibit strong stereotypical alignments when facts are missing.
  • Language-Dependent Bias: Bias is not a static model property; it fluctuates across languages. Gemma 2 9B showed increased stereotype alignment in Bengali compared to English, while Mistral-Nemo showed a decrease.
  • Contextual Sensitivity: Disambiguated contexts (where the answer is clear) led to much higher accuracy gains in Bengali (+36.59 points) than in English (+17.71 points), suggesting that models rely more heavily on explicit hints in lower-resource languages.

Key Results

The authors evaluated eight models, including Gemma 2, Llama 3.1, Mistral Nemo, and Qwen 2.5.

  • Top Performer: Gemma 2 9B achieved the highest overall accuracy in both English (93.44%) and Bengali (84.68%).
  • Cross-Lingual Gap: All models performed worse in Bengali than in English. Mistral-Nemo 12B demonstrated the most "stable" cross-lingual performance with the smallest accuracy gap (5.12 points).
  • Category-Specific Bias: Bias varies wildly by topic. For example, Qwen 2.5 7B showed significantly higher bias in the "Politics" and "Physical Appearance" categories when prompted in Bengali compared to English.
Table III: Accuracy comparison across models and languages.
Table VI: Selected category-level bias scores showing higher bias in Bengali for certain topics.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Naturalness: Future iterations need deeper checks on the linguistic "naturalness" of the Bengali sentences to ensure they don't contain accidental contextual hints.
  • Scope: The dataset could be expanded to include even more social scenarios and topics specific to Bangladeshi society.
  • Prompt Sensitivity: As seen in their experiments with BanglaLLM, prompt formatting (chat vs. completion) significantly impacts results. Standardizing these formats across all models is a goal for future comparative studies.
  • Human Review: While structural validation is automated, more extensive independent human review of the entire 61k+ record set would further increase cultural specificity and reliability.

Summary updated

Abstract

Despite growing attention to social bias in large language models (LLMs), existing bias benchmarks provide limited coverage of culturally grounded evaluation in Bengali, particularly within Bangladesh's sociocultural context. To address this gap, we develop a bilingual extension of BanglaBBQ comprising 61,312 question-answering instances across 12 social categories in English and Bengali. The benchmark incorporates balanced answer permutations, contrasting question framings, and both ambiguous and disambiguated contexts to systematically examine model performance and stereotype-aligned bias across languages. We validate the dataset through structural checks, 220 statistical tests, and expert annotation of 150 instances by four annotators, achieving strong inter-annotator agreement (Fleiss' \k{appa}=0.8349). We evaluate eight LLMs across 490,496 instances and identify five key findings. First, disambiguated contexts yield substantially greater accuracy improvements in Bengali (+36.59 percentage points) than in English (+17.71). Second, all evaluated models exhibit lower accuracy in Bengali, although Mistral-Nemo shows the smallest cross-lingual accuracy gap (5.12 percentage points). Third, stereotype-aligned bias exhibits language and model dependent variation, increasing in Bengali for Gemma 2 9B but decreasing for Mistral-Nemo. Fourth, higher accuracy does not necessarily correspond to lower stereotype-aligned bias. Finally, bias varies considerably across social categories, with Qwen 2.5 7B exhibiting stronger stereotype alignment in Bengali for Politics and Physical Appearance. These findings demonstrate that cross-lingual performance and social bias are distinct dimensions of LLM behavior and underscore the importance of culturally grounded, language-sensitive benchmarks for evaluating fairness beyond English-centric settings.
BibTeX
@misc{saha2026bdbbq,
  author = {Provat Saha and Ashika Habib Khan and Tahseen Tanha and Hasin Ishrak and Muhammad Iqbal Hossain},
  title = {{BDBBQ: A Human-AI Collaborative Benchmark for Bengali Social Bias in LLMs}},
  year = {2026},
  eprint = {2611.12407},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12407},
}
Figure