CausalDialogue: Modeling Utterance-level Causality in Conversations

Yi-Lin Tuan, Alon Albalak, Wenda Xu, Michael Saxon, Connor Pryor, Lise Getoor, William Yang Wang

Abstract

Despite the advancements in Large Language Models (LLMs), conversational agents frequently produce generic, uninspired responses. This paper identifies a lack of "agility"—the ability to provide nuanced, diverse responses tailored to subtle changes in dialogue history—as a primary cause. To address this, we introduce CausalDialogue, a novel dataset that models conversations as Directed Acyclic Graphs (DAGs) rather than linear sequences. We further propose ExMATE, a causality-inspired loss function, and CCE, a new metric to quantify agility. Our results demonstrate that modeling utterance-level causality significantly improves a model's ability to generate diverse and context-sensitive responses.

Introduction

A common failure mode for modern chatbots is the "generic response" problem. When presented with slightly varying prompts, models often converge on safe but uninformative answers like "I don't know." The authors argue this stems from traditional training objectives (Maximum Likelihood Estimation) that treat dialogue as a linear progression, failing to account for the branching nature of human interaction. This paper introduces the concept of Agility: the capacity of a model to produce distinct future turns when the immediate preceding turn changes, even if the broader context remains constant.

The CausalDialogue Dataset

The core contribution of this work is the CausalDialogue dataset. Unlike previous datasets that are strictly linear or tree-structured, CausalDialogue is structured as a Directed Acyclic Graph (DAG). This structure allows for two critical conversational phenomena:

  1. Branch-Splitting (Forks): A single utterance that can naturally lead to multiple distinct responses.
  2. Branch-Colliding (Colliders): Different conversational paths that eventually converge on the same response.
A conversational DAG illustrating how a single utterance can lead to multiple responses (forks) or how different paths can converge (colliders).

The dataset was seeded using expert-written scripts from the role-playing game Fire Emblem: Three Houses, which features inherent branching dialogue. These seeds were expanded via Amazon Mechanical Turk, resulting in a robust dataset of 2,322 dialogues and over 46,000 utterances.

Dataset Comparison

When compared to existing benchmarks like DailyDialog or PersonaChat, CausalDialogue is unique in its combination of DAG structure, situated profiles, and expert-involved quality control.

CausalDialogue is the only dataset providing a DAG structure with expert-involved quality and situated profiles.

Methodology

ExMATE Loss Function

To leverage the DAG structure, the authors developed ExMATE (Exponential Maximum Average Treatment Effect). Inspired by causal inference in medicine, ExMATE treats a specific utterance as a "treatment." The loss function is designed to maximize the probability of a response given its actual cause (the treatment) while minimizing its probability given a "counterfactual" or mismatched cause. This encourages the model to learn the specific causal link between a prompt and its unique response, rather than averaging probabilities across all possible futures.

CCE Metric

To measure agility, the researchers introduced CCE (Confidence Causal-Effect). This metric calculates the difference in perplexity when a model is presented with a correct cause-effect pair versus a mismatched one. A higher CCE score indicates that the model is more sensitive to the specific nuances of the preceding utterance.

Key Results and Insights

  • Enhanced Agility: Models trained with ExMATE showed significant improvements in the CCE metric. For instance, using a T5 backbone, ExMATE increased the CCE score from 1.39 to 1.50.
  • Diversity and Informative Responses: ExMATE consistently improved "Distinct-n" scores, indicating a reduction in generic outputs. Human evaluators also rated ExMATE-trained models as more informative and agile.
  • Architecture Performance: The T5 (encoder-decoder) architecture generally outperformed DialoGPT (decoder-only) in maintaining "Identity Accuracy," or staying in character throughout the dialogue.

Conclusion and Future Work

CausalDialogue and the ExMATE loss function provide a framework for moving beyond generic conversational AI. By framing utterances as causes, models can learn to navigate complex, branching interactions more effectively. While the current dataset is of moderate scale and limited to English, it serves as a powerful testbed for future research, particularly in the realm of offline Reinforcement Learning (RL), where models could learn to navigate conversational paths toward specific goals.

Summary updated

Abstract

Despite their widespread adoption, neural conversation models have yet to exhibit natural chat capabilities with humans. In this research, we examine user utterances as causes and generated responses as effects, recognizing that changes in a cause should produce a different effect. To further explore this concept, we have compiled and expanded upon a new dataset called CausalDialogue through crowd-sourcing. This dataset includes multiple cause-effect pairs within a directed acyclic graph (DAG) structure. Our analysis reveals that traditional loss functions struggle to effectively incorporate the DAG structure, leading us to propose a causality-enhanced method called Exponential Maximum Average Treatment Effect (ExMATE) to enhance the impact of causality at the utterance level in training neural conversation models. To evaluate the needs of considering causality in dialogue generation, we built a comprehensive benchmark on CausalDialogue dataset using different models, inference, and training methods. Through experiments, we find that a causality-inspired loss like ExMATE can improve the diversity and agility of conventional loss function and there is still room for improvement to reach human-level quality on this new dataset.
BibTeX
@inproceedings{tuan2023causaldialogue,
  author = {Yi-Lin Tuan and Alon Albalak and Wenda Xu and Michael Saxon and Connor Pryor and Lise Getoor and William Yang Wang},
  title = {{CausalDialogue: Modeling Utterance-level Causality in Conversations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=tL_so7vNUW5},
}
Figure