How did we get here? Summarizing conversation dynamics

Yilun Hua, Nicholas Chernogor, Yuzhe Gu, Seoyeon Julie Jeong, Miranda Luo, Cristian Danescu-Niculescu-MizilNAACL 2024 · DOI

TL;DR

This paper introduces a new task: summarizing the dynamics of conversations, not just their content. They show that these "conversation dynamics summaries" (SCDs) significantly improve both human and automated systems' ability to predict if a conversation will derail into toxic behavior, making forecasting faster and more accurate.

What Problem Does This Paper Address?

Conversations happen on two levels: the explicit content (what's being discussed) and the implicit dynamics (how participants interact, their tone, strategies, etc.). While traditional summarization focuses on content, understanding the dynamics is crucial for grasping a conversation's trajectory and predicting its future. This is particularly important for applications like online moderation, where identifying potentially problematic conversations quickly is key. Existing methods struggle to capture and synthesize these dynamic aspects concisely, leaving a gap in understanding and predicting conversation outcomes.

Methods

The authors introduce a novel task and a procedure for creating "Summaries of Conversation Dynamics" (SCDs). This involves a two-part human annotation process:

  1. Individual Work: An annotator drafts a comprehensive summary, identifying key points, turning points, and elements of conversation dynamics, then condenses it and writes speaker-specific summaries.
  2. Interactive Work: A second annotator, who hasn't read the transcript, reviews the first annotator's work and asks clarifying questions to ensure the summary accurately captures the conversation's trajectory and dynamics.

They also explore machine-generated SCDs using prompts for large language models (LLMs) like GPT-3.5-turbo, comparing traditional summarization prompts with more specialized "zeroshot" and "procedural" prompts designed to elicit dynamic information.

Datasets

The primary dataset used is the Conversations Gone Awry - ChangeMyView (CGA-CMV) dataset. This dataset consists of paired conversations: one that derailed into toxic behavior and one that did not, on the same topic. The authors created a new dataset of 50 human-written SCDs for conversations from the CGA-CMV training split.

Key Insights and Ideas

  • Dynamics Matter for Prediction: The core idea is that how a conversation unfolds (dynamics) is a strong predictor of its future, even more so than just the topic.
  • Human-AI Collaboration for Summaries: The two-stage human annotation process highlights the difficulty of capturing dynamics and the value of interactive refinement.
  • LLMs Can Learn Dynamics (with the right prompts): While LLMs can generate summaries, specific prompts are needed to guide them towards capturing conversational dynamics rather than just topical content.
  • SCDs Enhance Interpretability: By providing a concise summary of why a conversation might derail, SCDs can make automated forecasting systems more transparent and trustworthy for human users.

Key Results

  • Human Forecasting: Humans could predict conversation derailment three times faster and with greater confidence when using SCDs compared to reading full transcripts.
  • Automated Forecasting: Machine-based forecasting systems achieved higher accuracy when using SCDs as an intermediate step compared to predicting directly from transcripts. The "procedural prompt" generated SCDs performed best, achieving 67.3% accuracy in predicting derailment, significantly outperforming transcript-based methods (around 56-60%).
  • Human vs. Machine Summaries: While machine-generated summaries were better at capturing topical content, human-written SCDs provided a richer understanding of conversation dynamics, leading to higher human confidence.

Limitations and Future Work

  • Human-Machine Gap: A significant gap remains between the quality of human-written SCDs and those generated by machines, motivating further research into specialized models.
  • Prompt Engineering Limitations: While procedural prompts improved LLM performance, they might not be sufficient to fully "untrain" LLMs from traditional summarization biases.
  • Real-time Deployment: The current work focuses on post-hoc summarization; future work needs to address real-time generation for ongoing conversations.
  • Multimodal Dynamics: The current work is text-based; future research could incorporate vocal cues and gestures for a more holistic understanding.
  • Ethical Considerations: The authors emphasize the need for careful consideration of fairness and bias when deploying SCDs, especially in sensitive applications like moderation.

Summary updated

BibTeX
@inproceedings{hua2024how,
  author = {Yilun Hua and Nicholas Chernogor and Yuzhe Gu and Seoyeon Julie Jeong and Miranda Luo and Cristian Danescu-Niculescu-Mizil},
  title = {{How did we get here? Summarizing conversation dynamics}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.414},
}
Figure