Abstract
Despite the advancements in Large Language Models (LLMs), conversational agents frequently produce generic, uninspired responses. This paper identifies a lack of "agility"—the ability to provide nuanced, diverse responses tailored to subtle changes in dialogue history—as a primary cause. To address this, we introduce CausalDialogue, a novel dataset that models conversations as Directed Acyclic Graphs (DAGs) rather than linear sequences. We further propose ExMATE, a causality-inspired loss function, and CCE, a new metric to quantify agility. Our results demonstrate that modeling utterance-level causality significantly improves a model's ability to generate diverse and context-sensitive responses.
Introduction
A common failure mode for modern chatbots is the "generic response" problem. When presented with slightly varying prompts, models often converge on safe but uninformative answers like "I don't know." The authors argue this stems from traditional training objectives (Maximum Likelihood Estimation) that treat dialogue as a linear progression, failing to account for the branching nature of human interaction. This paper introduces the concept of Agility: the capacity of a model to produce distinct future turns when the immediate preceding turn changes, even if the broader context remains constant.
The CausalDialogue Dataset
The core contribution of this work is the CausalDialogue dataset. Unlike previous datasets that are strictly linear or tree-structured, CausalDialogue is structured as a Directed Acyclic Graph (DAG). This structure allows for two critical conversational phenomena:
- Branch-Splitting (Forks): A single utterance that can naturally lead to multiple distinct responses.
- Branch-Colliding (Colliders): Different conversational paths that eventually converge on the same response.
The dataset was seeded using expert-written scripts from the role-playing game Fire Emblem: Three Houses, which features inherent branching dialogue. These seeds were expanded via Amazon Mechanical Turk, resulting in a robust dataset of 2,322 dialogues and over 46,000 utterances.
Dataset Comparison
When compared to existing benchmarks like DailyDialog or PersonaChat, CausalDialogue is unique in its combination of DAG structure, situated profiles, and expert-involved quality control.
Methodology
ExMATE Loss Function
To leverage the DAG structure, the authors developed ExMATE (Exponential Maximum Average Treatment Effect). Inspired by causal inference in medicine, ExMATE treats a specific utterance as a "treatment." The loss function is designed to maximize the probability of a response given its actual cause (the treatment) while minimizing its probability given a "counterfactual" or mismatched cause. This encourages the model to learn the specific causal link between a prompt and its unique response, rather than averaging probabilities across all possible futures.
CCE Metric
To measure agility, the researchers introduced CCE (Confidence Causal-Effect). This metric calculates the difference in perplexity when a model is presented with a correct cause-effect pair versus a mismatched one. A higher CCE score indicates that the model is more sensitive to the specific nuances of the preceding utterance.
Key Results and Insights
- Enhanced Agility: Models trained with ExMATE showed significant improvements in the CCE metric. For instance, using a T5 backbone, ExMATE increased the CCE score from 1.39 to 1.50.
- Diversity and Informative Responses: ExMATE consistently improved "Distinct-n" scores, indicating a reduction in generic outputs. Human evaluators also rated ExMATE-trained models as more informative and agile.
- Architecture Performance: The T5 (encoder-decoder) architecture generally outperformed DialoGPT (decoder-only) in maintaining "Identity Accuracy," or staying in character throughout the dialogue.
Conclusion and Future Work
CausalDialogue and the ExMATE loss function provide a framework for moving beyond generic conversational AI. By framing utterances as causes, models can learn to navigate complex, branching interactions more effectively. While the current dataset is of moderate scale and limited to English, it serves as a powerful testbed for future research, particularly in the realm of offline Reinforcement Learning (RL), where models could learn to navigate conversational paths toward specific goals.
Summary updated