Abstract
Social norms are the shared rules that govern human interaction, yet they remain a significant challenge for Natural Language Processing (NLP) models due to their situational and cultural variability. This paper presents a novel pipeline developed by researchers at Purdue University that integrates Large Language Models (LLMs) with human-in-the-loop expertise to ground AI in cultural social norms. By utilizing a structured Cultural Context Schema, the authors generated a large-scale dataset of social norm descriptions for Chinese conversations. Experimental results demonstrate that providing explicit cultural context significantly enhances model performance on downstream tasks such as emotion detection and dialogue act recognition.
The Challenge of Cultural Grounding
While humans navigate social nuances effortlessly—knowing, for instance, when to use a formal title versus a first name—NLP models often struggle with these subtleties. Social norms are highly situational and vary significantly across cultures. Existing attempts to incorporate cultural awareness into AI face three primary hurdles:
- Data Scarcity: Most datasets are small or rely on synthetic, "robotic" conversations.
- Reliability: LLMs are prone to hallucinations or sensitivity to prompt phrasing.
- Lack of Structure: Previous work often treated norms as flat text rather than structured concepts tied to specific social roles.
To address these, the authors tackle Cultural Context Grounding: the process of representing, inducing, and verifying social norms within real-world conversational data at scale.
Methodology: The LLM-Human Pipeline
The proposed pipeline balances the raw generative power of LLMs with the precision of human cultural experts across five distinct stages.
- LLM Description Generation: Initial norm and violation descriptions are generated using GPT-3.5 for approximately 23,000 real Chinese conversations.
- Human-in-the-Loop (HiL) Concept Discovery: Human experts group these descriptions into 35 high-level "Norm Concepts" (e.g., "Respect for Authority" or "Filial Piety"). A k-Nearest Neighbors (k-NN) approach is used to scale these judgments across the dataset.
- Symbolic Grounding: LLMs act as symbolic annotators to map specific conversation turns to these high-level concepts, identifying the "actor" and "recipient" of the norm.
- Automated Verification (AgentEval): To clean noisy LLM data, a multi-agent system is employed. One agent acts as a critic to set evaluation criteria, another as a quantifier to grade the data, and a third as a verifier to ensure robustness.
- Downstream Task Evaluation: The grounded data is used to train models for specific conversational tasks, such as emotion detection.
The Cultural Context Schema
Central to this approach is a structured schema that separates factual context from cultural understanding. This separation allows models to perform more complex pragmatic reasoning by understanding not just who is talking, but what is expected of them.
Datasets and Experimental Setup
The study utilized three primary Chinese-language conversational datasets: MPDD (TV series scripts), CPED (multi-modal TV show data), and LDC CCU ZH Text (online forums and chats). From these, the researchers created a new corpus of 63,000 norm descriptions grounded in 23,500 real conversations.
To test the efficacy of this grounded data, the researchers employed a specialized graph model, ConvGraph, to evaluate performance on downstream tasks.
Key Results
Integrating cultural context provided a clear empirical advantage across multiple benchmarks:
- Emotion Detection (MPDD): The F1 score improved from 61.13 (without context) to 64.34 (with cultural context).
- Dialogue Act Detection (LDC CCU): Performance increased significantly from 64.84 to 71.74 F1.
- LLM Augmentation: Even advanced models like Llama-3.1-8B saw gains when augmented with this schema, suggesting that pre-trained cultural knowledge is insufficient without explicit, grounded context.
Discussion and Limitations
While the pipeline is generalizable, the current data is specific to Chinese culture. The authors acknowledge potential Western biases in the underlying LLMs (like GPT-3.5) and the significant labor costs associated with the human-in-the-loop stage. Future work will focus on expanding the schema to include more languages and exploring how grounded norms can improve AI behavior in multi-cultural social settings.
Summary updated