TL;DR
AfriNORM is a new evaluation benchmark designed to test whether Large Language Models (LLMs) actually understand the complex social norms and taboos—the "deontic modality"—of African cultures. By testing models on Yorùbá cultural scenarios in both English and Yorùbá, the researchers found that while models appear culturally competent in English, they suffer from significant "semantic confusion" when reasoning in the local language.
What Problem Does This Paper Address?
As LLMs are increasingly deployed in African sectors like health and finance, "cultural alignment" becomes a safety issue. Most current benchmarks measure linguistic fluency (can the model speak Yorùbá?) or general social conventions (what do people usually do?). However, they fail to capture deontic modality: the strict rules about what one should or must not do within a specific culture.
In many African contexts, failing to follow a social convention might be awkward, but violating a cultural taboo can lead to deep offense or social exclusion. The authors argue that a model might be fluent in a language but still lack the "lived epistemology"—the interwoven system of communal practices and spiritual traditions—required to provide safe and appropriate guidance.
Methods
The authors developed a structured pipeline to create a scenario-based, multiple-choice question (MCQ) dataset. The methodology follows five key steps:
- Scenario Curation: Local experts craft culturally grounded social scenarios.
- Adversarial Distractor Generation: For each scenario, three specific types of answers are created to probe the model's reasoning (see Figure 2):
- Option A (Correct Cultural Norm): The culturally appropriate response.
- Option B (Commonsense Default): A "deculturalized," neutral answer that might be correct in a Western or global context but misses the specific African norm.
- Option C (Incorrect Cultural Distractor): A plausible-sounding but incorrect answer that blends elements of related cultural scenarios to trick the model.
- Dual-Language Setup: Scenarios are provided in both English and the target African language (Yorùbá in this pilot) to compare performance across linguistic contexts.
- Zero-Shot Prompting: Models are prompted without specific cultural tuning to observe their natural biases.
- Metric Computation: Beyond simple accuracy, the paper introduces two diagnostic metrics:
- Universal Commonsense Bias Score (UCBS): How often the model defaults to the "neutral" global answer.
- Distractor Bias Score (DBS): How often the model is fooled by the "made-up" cultural distractor, indicating genuine confusion.
Datasets
The paper introduces the AfriNORM Yorùbá Dataset, a pilot benchmark consisting of 65 high-quality, expert-curated questions. These questions cover a wide range of cultural domains, including Respect, Dietary Permissions, Kingship, and Life Cycle/Metaphysics.
The authors compare AfriNORM to existing benchmarks like AFROBENCH and BLEND, highlighting that AfriNORM is unique in its specific focus on deontic modality (norms and taboos) rather than just general reasoning or social conventions.
Key Insights and Ideas
- The "Language-Culture Gap": The most striking finding is the performance delta between English and Yorùbá prompts. Models seem to "know" the cultural rules when asked in English but lose that knowledge when reasoning in the native language.
- Semantic Confusion vs. Simple Bias: The researchers found that the performance drop in Yorùbá wasn't just because models were defaulting to "Western" answers (UCBS remained relatively stable). Instead, the Distractor Bias Score (DBS) spiked, suggesting that the models experience "semantic confusion"—they recognize the context is cultural but can no longer distinguish between a real norm and a plausible-sounding fake one.
- Adversarial Design Matters: By including a "Commonsense Default," the researchers can distinguish between a model that is culturally ignorant (choosing the neutral answer) and one that is culturally confused (choosing the distractor).
Key Results
The study evaluated three major models: GPT-5, Gemini 2.0 Flash, and Llama 3.1 8B Instruct.
- High Performance in English: All models showed strong cultural understanding when prompted in English, with GPT-5 leading at 98.46% accuracy.
- Significant Degradation in Yorùbá: When switched to Yorùbá, accuracy dropped across the board. Llama 3.1 was hit hardest, falling to 52.31%—barely above random chance.
- Confusion Spikes: For Llama 3.1, the Distractor Bias Score (DBS) jumped from 4.69% in English to a staggering 38.46% in Yorùbá, confirming that the model becomes highly susceptible to "fake" cultural norms when operating in the local language.
Limitations and Future Work
The authors acknowledge two primary limitations:
- Scale: The pilot dataset is small (65 questions), which may limit the statistical power of the bias quantification.
- Scope: The current validation is limited to the Yorùbá language.
Future Work: The team plans to expand AfriNORM to include 20 other African languages and cultures. They also suggest investigating whether advanced prompting techniques like Chain-of-Thought (CoT) can bridge this normative reasoning gap or if the models require more fundamental alignment with African deontic modalities during training.
Summary updated