RENOVI: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversations
Lists
Only you see this
Abstract
Norm violations occur when individuals fail to conform to culturally accepted behaviors, which may lead to potential conflicts. Remediating norm violations requires social awareness and cultural sensitivity of the nuances at play. To equip interactive AI systems with a remediation ability, we offer ReNoVi — a large-scale corpus of 9,258 multi-turn dialogues annotated with social norms, as well as define a sequence of tasks to help understand and remediate norm violations step by step. ReNoVi consists of two parts: 512 human-authored dialogues (real data), and 8,746 synthetic conversations generated by ChatGPT through prompt learning. While collecting sufficient human-authored data is costly, synthetic conversations provide suitable amounts of data to help mitigate the scarcity of training data, as well as the chance to assess the alignment between LLMs and humans in the awareness of social norms. We thus harness the power of ChatGPT to generate synthetic training data for our task. To ensure the quality of both human-authored and synthetic data, we follow a quality control protocol during data collection. Our experimental results demonstrate the importance of remediating norm violations in socio-cultural conversations, as well as the improvement in performance obtained from synthetic data.
BibTeX
@inproceedings{zhan2024renovi,
author = {Haolan Zhan and Zhuang Li and Xiaoxi Kang and Tao Feng and Yuncheng Hua and Lizhen Qu and Yi Ying and Mei Rianto Chandra and Kelly Rosalin and Jureynolds Jureynolds and Suraj Sharma and Shilin Qu and Linhao Luo and Ingrid Zukerman and Lay-Ki Soon and Zhaleh Semnani-Azad and Gholamreza Haffari},
title = {{RENOVI: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversations}},
booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
year = {2024},
doi = {10.18653/v1/2024.findings-naacl.196},
}