MTP: A Dataset for Multi-Modal Turning Points in Casual Conversations

Gia-Bao Dinh Ho, Chang Wei Tan, Zahra Zamanzadeh Darban, Mahsa Salehi, Reza Haffari, Wray L. Buntine

Introduction

Human conversations are dynamic and non-linear, often marked by sudden shifts in tone, perspective, or intent. Identifying these "Turning Points" (TPs) is a complex task for artificial intelligence, requiring deep social reasoning beyond simple highlight detection. A Turning Point is defined as a moment triggered by an identifiable event that significantly alters a participant's subjective state—including feelings, behaviors, perspectives, or decisions.

An example of a Turning Point (TP) where a character's emotional state shifts from happy to sad based on conversational context.

The MTP Dataset

To address the lack of nuanced social reasoning datasets, the authors introduced the Multimodal Turning Points (MTP) dataset. Sourced from the first four seasons of The Big Bang Theory, the dataset leverages the show's eccentric characters and frequent social friction to provide a rich source of TPs.

The dataset comprises 340 conversation videos totaling 13.3 hours, segmented into 12,351 utterance-level clips. Each video was annotated by three humans to ensure high consensus on TP timestamps, causal explanations, and the "before vs. after" states of the characters involved. The authors categorized TPs into five distinct types: Emotional Outbursts, Changes in Decisions, External Influences, Shifts in Perspective, and Uncomfortable Situations.

TPMaven: A Multi-Modal Reasoning Framework

The authors proposed TPMaven, a two-stage framework designed to detect and explain these shifts:

  1. Scene Describer: Using LLaVA-7B, the system generates detailed descriptions of every utterance, capturing non-verbal cues like facial expressions and gestures. These are then summarized by GPT-3.5 into a concise visual narrative.
  2. Reasoner: The visual narrative, transcript, and speaker IDs are processed by GPT-4. A specialized prompting strategy is employed, featuring Tracking Instructions (maintaining a state for each speaker) and Chain-of-Thought Reasoning to identify the cause of a shift before pinpointing its timestamp.

Experimental Results

The framework was evaluated on two tasks: Turning Point Classification (MTPC) and Turning Point Detection (MTPD).

Performance metrics for Turning Point classification and detection across different model configurations.

As shown in the results, GPT-4 with few-shot prompting and tracking instructions significantly outperformed other configurations. In the classification task, it achieved an F1-score of 0.88. For the more challenging detection task (pinpointing the TP within a 20-second window), the model reached an F1-score of 0.61. The "tracking prompt" proved essential, as it forced the model to notice when character states diverged from their established baseline.

Discussion and Future Work

While TPMaven shows promise, several challenges remain. The current dataset is culturally specific to Western social norms, and the model primarily focuses on TPs occurring within verbal utterances. Future research could expand to non-verbal TPs (e.g., during silences) and cross-cultural contexts. Additionally, while the model generates explanations for TPs, automated evaluation of reasoning quality remains an open problem.

This work represents a significant step toward AI that understands the emotional and social nuances of human interaction, with potential applications in therapy, negotiation, and human-computer interfaces.

Summary updated

Abstract

Detecting critical moments, such as emotional outbursts or changes in decisions during conversations, is crucial for understanding shifts in human behavior and their consequences. Our work introduces a novel problem setting focusing on these moments as turning points (TPs), accompanied by a meticulously curated, high-consensus, human-annotated multi-modal dataset. We provide precise timestamps, descriptions, and visual-textual evidence high-lighting changes in emotions, behaviors, perspectives, and decisions at these turning points. We also propose a framework, TPMaven, utilizing state-of-the-art vision-language models to construct a narrative from the videos and large language models to classify and detect turning points in our multi-modal dataset. Evaluation results show that TPMaven achieves an F1-score of 0.88 in classification and 0.61 in detection, with additional explanations aligning with human expectations.
BibTeX
@inproceedings{ho2024mtp,
  author = {Gia-Bao Dinh Ho and Chang Wei Tan and Zahra Zamanzadeh Darban and Mahsa Salehi and Reza Haffari and Wray L. Buntine},
  title = {{MTP: A Dataset for Multi-Modal Turning Points in Casual Conversations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-short.30},
}
Figure