DIWALI - Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context

Pramit Sahoo, Maharaj Brahma, Maunendra Sankar DesarkarEMNLP 2025

TL;DR

Large Language Models (LLMs) struggle with cultural adaptation, often producing biased or irrelevant outputs due to a lack of cultural knowledge. This paper introduces DIWALI, a novel, comprehensive dataset of Indian cultural concepts, and uses it to evaluate LLMs' ability to adapt text to Indian cultural contexts, revealing significant limitations in their current capabilities.

What Problem Does This Paper Address?

LLMs, despite their vast knowledge, often fail to align with specific cultural norms, leading to biased generations and a lack of cultural competence. This is particularly challenging for evaluating and improving LLMs for cultural adaptation, as existing datasets are often limited in scope, lack regional granularity, and may contain inaccuracies. The authors highlight that this deficiency stems from LLMs being predominantly trained on Western-centric data, making them ill-equipped to handle the rich diversity of cultures like India's, which boasts numerous languages, dialects, and traditions across its states and sub-regions. The paper aims to address this by creating a more robust dataset and systematically evaluating LLMs' performance on cultural text adaptation tasks within the Indian context.

Methods

The core of this work involves two main components: dataset creation and LLM evaluation.

Dataset Creation (DIWALI): The authors constructed a novel dataset called DIWALI, which comprises approximately 8,817 culture-specific items (CSIs) across 17 cultural facets, covering 36 sub-regions of India. This dataset was built by leveraging GPT-40 and extensive web searches, focusing on official cultural and tourism websites for authenticity. Each CSI includes the concept, a description, the sub-region, and a source link. The dataset was manually inspected and validated to ensure quality and cultural relevance.

LLM Evaluation: The authors evaluated seven open-weight LLMs from three families (Llama, Mistral, Gemma) on cultural text adaptation. They employed a two-pronged evaluation strategy:

  1. Automatic Evaluation: This involved using an "Adaptation Score" based on exact and fuzzy matching of replaced concepts against ground-truth CSIs from DIWALI, CANDLE, and DOSA datasets.
  2. LLM-as-Judge: Two LLMs (Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3) were used to score adapted texts on cultural relevance, language fluency, and mathematical integrity using a Likert scale.
  3. Human Evaluation: Fifty adapted generations per model were randomly sampled and evaluated by five human annotators from diverse socio-demographic backgrounds in India, focusing on cultural relevance.

Datasets

  • DIWALI Dataset: A newly introduced, comprehensive dataset of Indian culture-specific items (CSIs). It contains ~8,817 concepts across 17 cultural facets and covers 36 sub-regions of India. This dataset aims to provide better regional granularity and cultural coverage compared to existing resources.
  • CANDLE: An existing dataset of culture-specific items. The authors note that CANDLE has limitations in its coverage of Indian culture and contains false positives.
  • DOSA: Another existing dataset focusing on Indian cultural artifacts, but with limited representation compared to DIWALI.
  • GSM8k and MGSM: Benchmarks for evaluating mathematical reasoning, adapted for cultural text adaptation tasks. These datasets were chosen because the educational domain often reflects socio-cultural values and contains culturally grounded entities.
  • DailyDialog and ROCStories: Datasets used for evaluating cultural adaptation in dialogue and story generation contexts, respectively.

Key Insights and Ideas

  • LLMs struggle with deep cultural adaptation: While LLMs can perform surface-level replacements (e.g., swapping names), they often fail to integrate cultural elements in a way that is deeply resonant and contextually appropriate. This suggests a gap between understanding individual cultural items and grasping the nuanced interplay of cultural practices, beliefs, and scenarios.
  • DIWALI is a more comprehensive benchmark: The DIWALI dataset significantly outperforms existing datasets like CANDLE in evaluating LLMs' cultural adaptation capabilities, especially for the Indian context. This highlights the importance of culturally rich, regionally specific datasets for accurate assessment.
  • LLM-as-Judge is prone to inflation: LLMs acting as judges tend to assign higher scores for cultural relevance than human evaluators, potentially due to overvaluing superficial changes rather than deep cultural integration. This suggests caution is needed when relying solely on LLM-based evaluation for nuanced tasks like cultural adaptation.
  • Sub-regional bias is prevalent: LLMs exhibit a strong bias towards certain sub-regions within India, often neglecting others, particularly in the Northeastern states. This indicates that LLMs tend to propagate dominant cultural views from their training data rather than representing the full spectrum of diversity.

Key Results

  • DIWALI significantly outperforms CANDLE: Across various LLMs and datasets (GSM8k, MGSM, DailyDialog, ROCStories), DIWALI consistently yielded higher adaptation scores compared to CANDLE, indicating its greater effectiveness in evaluating cultural adaptation. For instance, on GSM8k with an English prompt, DIWALI showed significantly higher fuzzy and exact match scores for most models.
  • LLMs show shallow adaptation: Human evaluation revealed that LLMs often perform "surface-level concept replacement" but fail to make deeper adaptations necessary for cultural resonance. For example, an LLM might replace "Tuesday" with "Diwali" but keep the associated scenario (e.g., "sells CDs") unchanged, lacking meaningful cultural connection.
  • LLM-as-Judge overestimates cultural relevance: LLM judges consistently assigned higher cultural relevance scores than human evaluators, with an approximate inflation of +0.5 to +2.5. This suggests LLMs might be overly sensitive to superficial changes.
  • Sub-regional coverage is uneven: Heatmaps visualizing adaptation coverage across Indian sub-regions revealed significant biases, with certain regions being overrepresented and others underrepresented, depending on the facet and the LLM.

Limitations and Future Work

Limitations: * Limited LLM scope: The study primarily evaluated widely used multilingual LLMs, not models specifically fine-tuned for Indian contexts. * Human evaluation sample size: Human evaluation involved five annotators, which might not fully capture the vast cultural diversity. * Surface vs. Deep adaptation analysis: The analysis of surface vs. deep adaptation was based on a limited number of examples.

Future Work: * Expand LLM evaluation to include country-specific LLMs. * Conduct human evaluations with a larger and more diverse set of annotators representing a wider range of socio-demographic backgrounds. * Perform more extensive analysis on the "aboutness" of cultural adaptations by involving a larger cultural population. * Develop LLMs that can grasp nuanced cultural contexts and perform deeper, more resonant adaptations.

Summary updated

BibTeX
@inproceedings{sahoo2025diwali,
  author = {Pramit Sahoo and Maharaj Brahma and Maunendra Sankar Desarkar},
  title = {{DIWALI - Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure