CausalDialogue: Modeling Utterance-level Causality in Conversations

Yi-Lin Tuan, Alon Albalak, Wenda Xu, Michael Saxon, Connor Pryor, Lise Getoor, William Yang Wang

TL;DR

This paper introduces CausalDialogue, a novel dataset structured as a Directed Acyclic Graph (DAG) to model branching conversational paths. To help neural dialogue models tailor their responses to subtle nuances in user inputs—a property the authors call Agility—they propose a causality-inspired loss function named ExMATE (Exponential Maximum Average Treatment Effect).

What Problem Does This Paper Address?

Despite massive scale, modern chatbots often suffer from the "generic response problem," frequently defaulting to vague answers like "I don't know" or "I'm not sure." A key reason is that standard training datasets treat conversations as linear sequences, ignoring the underlying cause-and-effect relationships of dialogue turns.

The authors address a specific capability they call Agility: if all prior turns in a conversation are identical, but the very last turn differs semantically, how well can a model tailor its future responses to that specific difference? Traditional models struggle with this because standard Maximum Likelihood Estimation (MLE) loss functions do not explicitly penalize a model for treating distinct inputs as if they cause the same generic output.

Methods

To explicitly model utterance-level causality, the authors look to causal inference. They treat a preceding utterance as a "treatment" (cause) and the generated response as the "outcome" (effect).

They adapt the classical statistical concept of the Average Treatment Effect (ATE) into a new training objective called Exponential Maximum Average Treatment Effect (ExMATE) loss.

Figure 4: The graphical model of fork-like DAG consid- ered in our proposed ExMATE loss.

As shown in the graphical model above, given a shared dialogue history ($DH$), an utterance $X_1$ should cause response $Y_1$, while an alternative utterance $X_2$ should cause $Y_2$. The ExMATE loss function modifies standard MLE by adding a contrastive penalty term:

$$L_{ExMATE} = -\log P_\theta(y|DH, x) + \exp(\log P_\theta(y|DH, x_c))$$

Here, $x_c$ is a counterfactual (alternative) utterance from a parallel branch in the dataset. The exponential term linearizes the gradient of the negative constraint, preventing it from destabilizing training while ensuring the model learns that $y$ is uniquely caused by $x$, not $x_c$.

The authors also introduce a new evaluation metric, Confidence Causal-Effect (CCE), which measures a model's agility by calculating the difference in perplexity when predicting a correct response versus an incorrect cross-branch response.

Datasets

The authors introduce the CausalDialogue dataset. Unlike traditional datasets, it is structured as a Directed Acyclic Graph (DAG) to capture two realistic conversational phenomena:

  1. Branch-Splitting (Forks): Where a single utterance elicits multiple distinct potential responses.
  2. Branch-Colliding (Colliders): Where different conversational paths eventually lead back to the same response.
Figure 1: A dialogue DAG example in the new dataset CausalDialogue. As the conversation progress, each utterance can be continued with multiple responses (branch-splitting; fork); meanwhile, the same root dialogue with different middle turns can be continued by the same response (branch-colliding; collider).

The dataset was built by:

  • Sourcing expert-written scripts from the role-playing game Fire Emblem: Three Houses, which naturally contains branching narratives and 41 distinct character speaker profiles.
  • Expanding these scripts via Amazon Mechanical Turk (MTurk) crowd-sourcing to increase linguistic diversity.

The final dataset contains 2,322 dialogues, 4,866 branches, and 46,109 utterances.

Key Insights and Ideas

  • Conversations are DAGs, not Trees: While some prior work modeled dialogues as trees, the inclusion of "branch-colliding" (where different topics converge back to a single consensus or reaction) means human dialogue is fundamentally a graph (DAG).
  • The Entropy Trap of MLE: In a branching dataset, a single context $(DH, x)$ can map to multiple valid references ($y$). Standard MLE training tries to hedge its bets by averaging these distributions, leading to high-entropy, generic outputs.
  • Causal Inference as Regularization: ExMATE acts as a targeted regularizer. By forcing the model to recognize what a specific utterance does not cause, it sharpens the model's probability distribution around agile, context-specific responses.

Key Results

The authors benchmarked two backbone architectures: DialoGPT (decoder-only) and T5 (encoder-decoder), comparing MLE against ExMATE across various decoding strategies (Greedy, Softmax, Top-K).

  • Improved Agility: Models trained with ExMATE consistently outperformed their MLE counterparts on the Agility metric (CCE). For example, DialoGPT's CCE score jumped from 2.33 to 2.68, and T5's score improved from 1.39 to 1.50.
  • Enhanced Diversity: ExMATE significantly boosted response diversity (measured by Distinct-1 and Distinct-2 n-gram scores) without degrading the fluency (Perplexity/BLEU) of the text.
  • Human Evaluation: Human judges rated ExMATE-trained models as more agile and equally informative compared to MLE models, though they noted a slight trade-off in absolute coherence.

Limitations and Future Work

  • Scale: The dataset is of moderate size, making it ideal for fine-tuning rather than training large language models from scratch.
  • Coherence Trade-off: Because ExMATE explicitly subtracts the likelihood of counterfactual examples (which are still natural sentences), it can occasionally compromise the strict grammatical or contextual coherence of the output.
  • Remaining Artifacts: Qualitative analysis showed that models still suffer from standard generation flaws like mode collapse during greedy search, semantic repetition, and "identity misplacement" (getting confused about which character profile they are role-playing).

Summary updated

Abstract

Despite their widespread adoption, neural conversation models have yet to exhibit natural chat capabilities with humans. In this research, we examine user utterances as causes and generated responses as effects, recognizing that changes in a cause should produce a different effect. To further explore this concept, we have compiled and expanded upon a new dataset called CausalDialogue through crowd-sourcing. This dataset includes multiple cause-effect pairs within a directed acyclic graph (DAG) structure. Our analysis reveals that traditional loss functions struggle to effectively incorporate the DAG structure, leading us to propose a causality-enhanced method called Exponential Maximum Average Treatment Effect (ExMATE) to enhance the impact of causality at the utterance level in training neural conversation models. To evaluate the needs of considering causality in dialogue generation, we built a comprehensive benchmark on CausalDialogue dataset using different models, inference, and training methods. Through experiments, we find that a causality-inspired loss like ExMATE can improve the diversity and agility of conventional loss function and there is still room for improvement to reach human-level quality on this new dataset.
BibTeX
@inproceedings{tuan2023causaldialogue,
  author = {Yi-Lin Tuan and Alon Albalak and Wenda Xu and Michael Saxon and Connor Pryor and Lise Getoor and William Yang Wang},
  title = {{CausalDialogue: Modeling Utterance-level Causality in Conversations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=tL_so7vNUW5},
}
Figure