Multi-Cultural Norm Base: Frame-based Norm Discovery in Multi-Cultural Settings

Viet Thanh Pham, Shilin Qu, Farhad Moghimifar, Suraj Sharma, Yuan-Fang Li, Weiqing Wang, Reza Haf2024 · DOI

Abstract

Sociocultural norms serve as guiding principles for personal conduct in social interactions within a particular society or culture. The study of norm discovery has seen significant development over the last few years, with various interesting approaches. However, it is difficult to adopt these approaches to discover norms in a new culture, as they rely either on human annotations or real-world dialogue contents. This paper presents a robust automatic norm discovery pipeline, which utilizes the cultural knowledge of GPT-3.5 Turbo (ChatGPT) along with several social factors. By using these social factors and ChatGPT, our pipeline avoids the use of human dialogues that tend to be limited to specific scenarios, as well as the use of human annotations that make it difficult and costly to enlarge the dataset. The resulting database—Multicultural Norm Base (MNB)—covers 6 distinct cultures, with over 150k sociocultural norm statements in total. A state-of-the-art Large Language Model (LLM), Llama 3, fine-tuned with our proposed dataset, shows remarkable results on various downstream tasks, outperforming models fine-tuned on other datasets significantly.

Introduction

Sociocultural norms are the "unwritten rules" of society—knowing when to bow, how to address a professor, or what topics are taboo at dinner. For AI to be truly helpful and safe in a global context, it must understand these nuances. However, building datasets of these norms is notoriously difficult. Existing approaches usually suffer from two primary issues: the Human Bottleneck, where relying on human annotators is expensive and slow, and the Dialogue Dilemma, where extracting norms from real-world chat logs limits the scope to common scenarios and specific languages.

To bypass these hurdles, this research introduces a scalable, automated pipeline to "mine" cultural knowledge already implicitly stored within Large Language Models (LLMs) by systematically probing them with structured social scenarios.

Methodology: The Frame-Based Discovery Pipeline

The authors propose a "frame-based" discovery pipeline that generates synthetic scenarios to probe an LLM's cultural knowledge. Instead of searching for norms in existing text, the system constructs "situational frames" using 10 key social factors, including Culture (e.g., Afghan, Chinese, Canadian), Social Relation (e.g., student-professor), Power Distance, Age, and Location.

The automated pipeline for sociocultural norm discovery, moving from frame construction to the final Multicultural Norm Base.

The pipeline consists of four main stages:

  1. Situational Frame Construction: Defining frames based on the 10 social factors.
  2. Frame Filtering: Removing invalid combinations (e.g., a toddler in a professional working relationship) using a fine-tuned RoBERTa model.
  3. Norm Extraction: Prompting ChatGPT with valid frames to generate "Rules of Thumb" (RoTs) following specific templates.
  4. Deduplication & Verification: Using BERT embeddings to remove duplicates and a secondary LLM-based verification step to ensure the accuracy and logic of the generated statements.

The Multi-Cultural Norm Base (MNB)

The primary output of this pipeline is the Multi-Cultural Norm Base (MNB). This dataset features 155,929 norm statements derived from 28,804 situational frames across six cultures: American, British, Canadian, Chinese, Indian, and Afghan. By using a consistent set of social factors across all cultures, the researchers created a framework where differences in norms become clearly visible and measurable.

Evaluation and Results

The researchers fine-tuned Llama 3 (8B) on the MNB dataset and compared its performance against the base model and models trained on other datasets like SOCIAL-CHEM-101.

Performance comparison of Llama 3 variants across various NLP metrics, showing the significant gains achieved by fine-tuning on the MNB dataset.

Key findings include:

  • Moral Reasoning (MIC): The MNB-tuned model achieved a ROUGE-1 score of 30.41, nearly doubling the performance of the base Llama 3 (15.53) and significantly outperforming models trained on other social rule datasets.
  • Etiquette Sensitivity (EtiCor): On the EtiCor benchmark, the MNB model achieved an overall F1 score of 80.99, outperforming the baseline (72.45) across all tested regions.
  • Human Validation: Native speakers from the six cultures rated the generated norms on a 1-5 Likert scale. Most cultures scored above 3.8 across criteria like relevance, correctness, and insightfulness, confirming that the synthetic norms are high-quality and culturally accurate.

Discussion and Limitations

While the MNB represents a significant step forward in automated norm discovery, the authors acknowledge certain limitations. Since the norms are generated by ChatGPT, they may reflect implicit biases or Western-centric viewpoints present in the training data. Additionally, societal norms are dynamic and evolve over time, whereas a static database like MNB may eventually become outdated. Future work will focus on expanding the cultural scope beyond the initial six cultures and developing more scalable methods for intrinsic evaluation by native speakers.

Summary updated

Abstract

Sociocultural norms serve as guiding principles for personal conduct in social interactions within a particular society or culture.The study of norm discovery has seen significant development over the last few years, with various interesting approaches.However, it is difficult to adopt these approaches to discover norms in a new culture, as they rely either on human annotations or real-world dialogue contents.This paper presents a robust automatic norm discovery pipeline, which utilizes the cultural knowledge of GPT-3.5 Turbo (ChatGPT) along with several social factors.By using these social factors and ChatGPT, our pipeline avoids the use of human dialogues that tend to be limited to specific scenarios, as well as the use of human annotations that make it difficult and costly to enlarge the dataset.The resulting database -Multicultural Norm Base (MNB) -covers 6 distinct cultures, with over 150k sociocultural norm statements in total.A state-of-the-art Large Language Model (LLM), Llama 3, fine-tuned with our proposed dataset, shows remarkable results on various downstream tasks, outperforming models fine-tuned on other datasets significantly.
BibTeX
@misc{pham2024multi,
  author = {Viet Thanh Pham and Shilin Qu and Farhad Moghimifar and Suraj Sharma and Yuan-Fang Li and Weiqing Wang and Reza Haf},
  title = {{Multi-Cultural Norm Base: Frame-based Norm Discovery in Multi-Cultural Settings}},
  year = {2024},
  doi = {10.18653/v1/2024.conll-1.3},
}
Figure