RENOVI: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversations

Haolan Zhan, Zhuang Li, Xiaoxi Kang, Tao Feng, Yuncheng Hua, Lizhen Qu, Yi Ying, Mei Rianto Chandra, Kelly Rosalin, Jureynolds Jureynolds, Suraj Sharma, Shilin Qu, Linhao Luo, Lay-Ki Soon, Zhaleh Semnani-Azad, Ingrid Zukerman, Gholamreza Haffari

Introduction

While Large Language Models (LLMs) have made significant strides in general alignment, they often lack the nuanced "social intelligence" required to navigate specific cultural contexts. A behavior considered standard in one culture may constitute a "norm violation" in another, potentially leading to social friction. Current research predominantly focuses on detecting these violations or aligning models with broad universal values. However, there remains a critical gap in remediation: the ability of an AI to assist users in rephrasing offensive statements or understanding underlying social rules. This paper introduces RENOVI, a benchmark focused on remediating norm violations within Chinese culture, where hierarchy, politeness markers, and indirectness play pivotal roles.

Methodology: The Remediation Pipeline

The researchers operationalized Expectancy Violation Theory (EVT) and Interaction Adaptation Theory (IAT) to create a structured four-step pipeline for handling social faux pas.

Figure 1: The RENOVI four-step pipeline for norm violation detection and remediation, illustrated with a dialogue example regarding a request for a seat change.

The pipeline consists of:

  1. Norm Violation Detection: Identifying if an utterance breaks a social norm and categorizing the interaction (e.g., a request).
  2. Impact Estimation: Determining the severity of the violation (High vs. Low Impact).
  3. Remediation Generation: Rephrasing the violating utterance to align with cultural expectations.
  4. Justification Generation: Explaining the rationale behind the fix to educate the user on the relevant social rule.

The RENOVI Dataset

The primary contribution is the RENOVI Dataset, comprising 9,258 multi-turn dialogues. To ensure both quality and scale, the authors employed a hybrid approach: 512 dialogues were meticulously authored by humans, while 8,746 were synthetically generated using ChatGPT with ontology-based prompts.

Figure 2: Distribution of norm categories across synthetic and human-authored data in the RENOVI dataset.

The dataset covers seven core social norms: apology, criticism, greeting, persuasion, request, leave-taking, and thanking. Each entry is annotated with situational attributes such as location and role relationships (e.g., boss/subordinate).

Key Insights and Results

The Synthetic Data Trap

A significant finding was that training models exclusively on synthetic data can degrade performance due to "domain shift." Synthetic dialogues tend to be more clustered and less diverse than human-authored ones. However, using a small amount of human data as an "anchor" alongside synthetic data significantly boosts model performance, suggesting that synthetic data acts as an effective amplifier when grounded in human examples.

Model Performance

  • Detection & Impact: Fine-tuned models like RoBERTa-zh achieved F1 scores of approximately 60% for detection and 78% for impact estimation when trained on the mixed dataset.
  • Remediation: ChatYuan, a Chinese-specific LLM, combined with a Pfeiffer adapter, proved most effective at producing concise remediations that matched human style and length.
  • Human Evaluation: Human judges rated LLM-generated justifications highly for relevance and informativeness, though they noted that models still occasionally struggle with the nuances of politeness in rephrasing.

Conclusion and Future Work

RENOVI provides a structured framework for AI to learn, correct, and explain social errors. While currently focused on Chinese culture, the authors intend to expand the benchmark to include Spanish, Latin, and Arabic contexts. By moving beyond simple detection toward active remediation, RENOVI paves the way for interactive systems that serve as true socio-cultural bridges in global communication.

Summary updated

BibTeX
@inproceedings{zhan2024renovi,
  author = {Haolan Zhan and Zhuang Li and Xiaoxi Kang and Tao Feng and Yuncheng Hua and Lizhen Qu and Yi Ying and Mei Rianto Chandra and Kelly Rosalin and Jureynolds Jureynolds and Suraj Sharma and Shilin Qu and Linhao Luo and Lay-Ki Soon and Zhaleh Semnani-Azad and Ingrid Zukerman and Gholamreza Haffari},
  title = {{RENOVI: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.196},
}
Figure