AfriNORM: An Evaluation Benchmark for Normative Reasoning of LLMs Across African Cultures

Ahmad Samuel Gali, Tadesse Destaw Belay, Idris Abdulmumin, Seid Muhie Yimam, Shamsuddeen Hassan MuhammadarXiv 2026 · arXiv

TL;DR

AfriNORM is a new evaluation benchmark designed to test whether Large Language Models (LLMs) actually understand the complex social norms and taboos—the "deontic modality"—of African cultures. By testing models on Yorùbá cultural scenarios in both English and Yorùbá, the researchers found that while models appear culturally competent in English, they suffer from significant "semantic confusion" when reasoning in the local language.

What Problem Does This Paper Address?

As LLMs are increasingly deployed in African sectors like health and finance, "cultural alignment" becomes a safety issue. Most current benchmarks measure linguistic fluency (can the model speak Yorùbá?) or general social conventions (what do people usually do?). However, they fail to capture deontic modality: the strict rules about what one should or must not do within a specific culture.

In many African contexts, failing to follow a social convention might be awkward, but violating a cultural taboo can lead to deep offense or social exclusion. The authors argue that a model might be fluent in a language but still lack the "lived epistemology"—the interwoven system of communal practices and spiritual traditions—required to provide safe and appropriate guidance.

Methods

The authors developed a structured pipeline to create a scenario-based, multiple-choice question (MCQ) dataset. The methodology follows five key steps:

  1. Scenario Curation: Local experts craft culturally grounded social scenarios.
  2. Adversarial Distractor Generation: For each scenario, three specific types of answers are created to probe the model's reasoning (see Figure 2):
  3. Option A (Correct Cultural Norm): The culturally appropriate response.
  4. Option B (Commonsense Default): A "deculturalized," neutral answer that might be correct in a Western or global context but misses the specific African norm.
  5. Option C (Incorrect Cultural Distractor): A plausible-sounding but incorrect answer that blends elements of related cultural scenarios to trick the model.
  6. Dual-Language Setup: Scenarios are provided in both English and the target African language (Yorùbá in this pilot) to compare performance across linguistic contexts.
  7. Zero-Shot Prompting: Models are prompted without specific cultural tuning to observe their natural biases.
  8. Metric Computation: Beyond simple accuracy, the paper introduces two diagnostic metrics:
  9. Universal Commonsense Bias Score (UCBS): How often the model defaults to the "neutral" global answer.
  10. Distractor Bias Score (DBS): How often the model is fooled by the "made-up" cultural distractor, indicating genuine confusion.
The AfriNORM Evaluation Benchmark Methodology

Datasets

The paper introduces the AfriNORM Yorùbá Dataset, a pilot benchmark consisting of 65 high-quality, expert-curated questions. These questions cover a wide range of cultural domains, including Respect, Dietary Permissions, Kingship, and Life Cycle/Metaphysics.

The authors compare AfriNORM to existing benchmarks like AFROBENCH and BLEND, highlighting that AfriNORM is unique in its specific focus on deontic modality (norms and taboos) rather than just general reasoning or social conventions.

Breakdown of Cultural Domains in the AfriNORM Yorùbá Dataset

Key Insights and Ideas

  • The "Language-Culture Gap": The most striking finding is the performance delta between English and Yorùbá prompts. Models seem to "know" the cultural rules when asked in English but lose that knowledge when reasoning in the native language.
  • Semantic Confusion vs. Simple Bias: The researchers found that the performance drop in Yorùbá wasn't just because models were defaulting to "Western" answers (UCBS remained relatively stable). Instead, the Distractor Bias Score (DBS) spiked, suggesting that the models experience "semantic confusion"—they recognize the context is cultural but can no longer distinguish between a real norm and a plausible-sounding fake one.
  • Adversarial Design Matters: By including a "Commonsense Default," the researchers can distinguish between a model that is culturally ignorant (choosing the neutral answer) and one that is culturally confused (choosing the distractor).

Key Results

The study evaluated three major models: GPT-5, Gemini 2.0 Flash, and Llama 3.1 8B Instruct.

  • High Performance in English: All models showed strong cultural understanding when prompted in English, with GPT-5 leading at 98.46% accuracy.
  • Significant Degradation in Yorùbá: When switched to Yorùbá, accuracy dropped across the board. Llama 3.1 was hit hardest, falling to 52.31%—barely above random chance.
  • Confusion Spikes: For Llama 3.1, the Distractor Bias Score (DBS) jumped from 4.69% in English to a staggering 38.46% in Yorùbá, confirming that the model becomes highly susceptible to "fake" cultural norms when operating in the local language.
Performances of Llama 3.1 8B Instruct, Gemini 2.0 Flash, and GPT-5 models on the dataset

Limitations and Future Work

The authors acknowledge two primary limitations:

  1. Scale: The pilot dataset is small (65 questions), which may limit the statistical power of the bias quantification.
  2. Scope: The current validation is limited to the Yorùbá language.

Future Work: The team plans to expand AfriNORM to include 20 other African languages and cultures. They also suggest investigating whether advanced prompting techniques like Chain-of-Thought (CoT) can bridge this normative reasoning gap or if the models require more fundamental alignment with African deontic modalities during training.

Summary updated

Abstract

Large language models (LLMs) are increasingly deployed across various sectors. However, current evaluation benchmarks are insufficient to assess the applicability of these models in local African contexts, as most focus on linguistic fluency (grammar/translation) rather than cultural competence. The few existing benchmarks that measure general knowledge or social conventions do not capture the African deontic modality, i.e., the strict norms and taboos that govern diverse social interactions. In this paper, we introduce AfriNORM, a scenario-based, multiple-choice question dataset designed to measure and diagnose the normative reasoning limitations of LLMs when applied to African sociocultural contexts. We introduce two metrics: a Universal Commonsense Bias Score (UCBS) to quantify commonsense bias in LLM and a Distractor Bias Score (DBS) to quantify an LLM's limited understanding of African cultural norms. We also conduct a case study on the Yorùbá language, identifying confusion gaps in two closed-source LLMs, "Gemini 2.0 Flash" and "GPT-5", and one open-source LLM, "Llama 3.1 8B Instruct" when prompted in English versus Yorùbá. While their accuracy in English is high, their performance in Yorùbá is much lower, attributable to cultural confusion rather than simple bias, indicating that the models exhibit semantic confusion when reasoning about norms in Yorùbá.
BibTeX
@misc{gali2026afrinorm,
  author = {Ahmad Samuel Gali and Tadesse Destaw Belay and Idris Abdulmumin and Seid Muhie Yimam and Shamsuddeen Hassan Muhammad},
  title = {{AfriNORM: An Evaluation Benchmark for Normative Reasoning of LLMs Across African Cultures}},
  year = {2026},
  eprint = {2611.22580},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22580},
}
Figure