Building Socio-culturally Inclusive Stereotype Resources with Community Engagement

Sunipa Dev, Jaya Goyal, Dinesh Tewari, Shachi Dave, Vinodkumar Prabhakaran

Introduction

Researchers have developed SPICE (Stereotype Pooling in India with Community Engagement), a new dataset containing over 2,000 stereotypes unique to the Indian socio-cultural context. By moving beyond Western-centric benchmarks and engaging directly with diverse local communities, the study reveals that popular large language models (LLMs) harbor significant, previously unmeasured biases related to caste, religion, and regional identity.

The Problem with Current Fairness Benchmarks

Most "fairness" evaluations for AI are heavily Western-centric. They focus on axes of disparity common in the US or Europe—such as Western concepts of race—while ignoring the socially situated harms prevalent in other parts of the world. In India, for example, stereotypes often revolve around caste, specific regional identities, and complex religious intersections. Standard benchmarks completely miss these nuances.

Furthermore, most existing stereotype datasets are built using researcher-curated word lists or LLM-generated content. This creates a "top-down" bias where niche or marginalized perspectives are excluded. If we cannot measure these local harms, we cannot mitigate them, leading to a gross underestimation of the risks AI poses to global populations.

A Bottom-Up, Participatory Approach

To capture authentic local nuances, the authors opted for a participatory, bottom-up approach. Instead of asking participants to validate a pre-written list of stereotypes, they conducted open-ended surveys.

  1. Community Engagement: The team recruited 655 participants from government-aided universities across four major regions of India (North, South, East, West). This recruitment strategy ensured representation across mandatory reservation quotas for Scheduled Castes (SC), Scheduled Tribes (ST), and Other Backward Classes (OBC).
  2. Open-Ended Collection: Participants provided free-form text examples of stereotypes they encounter in society, parsing these into "identity terms" (who it is about) and "attribute terms" (the generalization).
Examples of stereotypes collected from different participant demographics.
  1. Data Pipeline: The raw text was manually cleaned and standardized (e.g., grouping "Punjabi" and "Punjabis") to create the final SPICE dataset, which features over 500 unique identity terms.
  2. NLI Evaluation: To test if AI models actually "believe" these stereotypes, researchers used a Natural Language Inference (NLI) framework. They presented models with a neutral premise (e.g., "The Punjabi person bought a cake") and a stereotypical hypothesis (e.g., "The alcoholic person bought a cake") to see if the model incorrectly "entailed" the association.

Key Insights

  • Fairness is Local: A model might pass a fairness test in California but fail miserably in Delhi. Stereotypes like "Jains are vegetarian" or "South Indians are intelligent" are culturally specific and require local knowledge to identify.
  • The Power of Free-Form: By allowing open-ended responses, the researchers uncovered 500 additional identity terms completely missing from previous dictionary-based or LLM-generated datasets.
  • Intersectional Nuance: The survey captured complex intersectional stereotypes (e.g., "Muslim women being oppressed" or "Kashmiri Pandits being victims") that simple word-association tests usually overlook.
  • The "Offensiveness Gap": The study found a massive disparity in how offensive stereotypes are perceived. Stereotypes about dominant groups (like Brahmins) were often viewed as "respected" or "neutral," while those about marginalized groups (Dalits, Muslims, or specific regional migrants) were highly derogatory (e.g., "criminal," "violent," "untouchable").

Key Results: Uncovering Hidden Model Bias

SPICE dramatically increased the visibility of known stereotypes in the Indian context, adding over 1,000 new entries compared to previous efforts. More importantly, it proved that existing models are far more biased than standard tests suggest.

When regional stereotypes from SPICE were added to evaluation pipelines, the measured bias in models like XLNet and ELECTRA nearly doubled.

Comparison of model entailment scores with and without the SPICE dataset.

As shown above, XLNet’s "entailment" (bias) score jumped from 7.5% to 16.0% when the more inclusive SPICE data was used. The researchers also found that many of these stereotypes (e.g., "Punjabi, loud" or "Brahmin, religious") co-occur hundreds of times in massive training datasets like C4 and Wikipedia, explaining exactly where models "learn" these associations.

Limitations and Future Work

While SPICE is a massive step forward, the authors note several limitations:

  • Language Barrier: The current study was conducted in English. Given India's 22 official languages, future work must expand to regional languages where stereotypes may be expressed differently.
  • Demographic Skew: Most participants were students aged 18–24. Older generations might hold different or more entrenched stereotypical views.
  • Geographic Gaps: Due to logistics, certain regions like North East India—which faces unique forms of marginalization—were underrepresented.

Finally, the authors acknowledge the dual-use nature of this work: a public list of stereotypes could be misused by bad actors. To mitigate this, strict guardrails and "data cards" are necessary for responsible usage.

Summary updated

Abstract

With rapid development and deployment of generative language models in global settings, there is an urgent need to also scale our measurements of harm, not just in the number and types of harms covered, but also how well they account for local cultural contexts, including marginalized identities and the social biases experienced by them. Current evaluation paradigms are limited in their abilities to address this, as they are not representative of diverse, locally situated but global, socio-cultural perspectives. It is imperative that our evaluation resources are enhanced and calibrated by including people and experiences from different cultures and societies worldwide, in order to prevent gross underestimations or skews in measurements of harm. In this work, we demonstrate a socio-culturally aware expansion of evaluation resources in the Indian societal context, specifically for the harm of stereotyping. We devise a community engaged effort to build a resource which contains stereotypes for axes of disparity that are uniquely present in India. The resultant resource increases the number of stereotypes known for and in the Indian context by over 1000 stereotypes across many unique identities. We also demonstrate the utility and effectiveness of such expanded resources for evaluations of language models. CONTENT WARNING: This paper contains examples of stereotypes that may be offensive.
BibTeX
@inproceedings{dev2023building,
  author = {Sunipa Dev and Jaya Goyal and Dinesh Tewari and Shachi Dave and Vinodkumar Prabhakaran},
  title = {{Building Socio-culturally Inclusive Stereotype Resources with Community Engagement}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
}
Figure