Improving Contextual Coherence in Variational Personalized and Empathetic Dialogue Agents

Jing Yang Lee, Kong Aik Lee, Woon-Seng Gan

Abstract

Latent variable models like Conditional Variational Autoencoders (CVAEs) are effective at generating diverse dialogue, but they often struggle to maintain contextual coherence. This paper introduces the Uncertainty-Aware CVAE (UA-CVAE), which models data-dependent uncertainty to keep responses on track. By incorporating the variance of the latent distribution into the generation process, the model produces responses that are both diverse and logically consistent. Furthermore, the authors propose a novel Utterance Entailment (UE) metric to automatically evaluate conversational coherence. Experimental results on standard datasets show that UA-CVAE significantly reduces nonsensical or contradictory replies while maintaining high diversity.

Introduction

In open-domain dialogue generation, agents must maintain a consistent persona and demonstrate empathy while avoiding generic, repetitive responses. While CVAEs address this by injecting stochasticity via a latent variable, this often comes at the cost of contextual coherence. Responses may be diverse but frequently ignore conversation history or assigned personas. The authors address this by making the CVAE "aware" of the uncertainty or noise in the input context, allowing for responses that are both varied and logically grounded.

Methodology

The proposed Uncertainty-Aware CVAE (UA-CVAE) framework leverages the observation that the variance ($\sigma^2$) of the latent Gaussian distribution in a CVAE approximates heteroscedastic aleatoric uncertainty—the specific ambiguity inherent in a given dialogue context.

UA-CVAE Architecture

The model utilizes a pre-trained GPT-2 to embed dialogue context and external information (e.g., persona or emotion). A prior network predicts the mean and variance of the latent distribution. Unlike standard CVAEs, the UA-CVAE employs a combination network to merge the sampled latent variable $z$ with the variance vector $\sigma^2$, creating an uncertainty-aware latent variable ($z_u$). This provides the decoder with explicit information regarding the input's uncertainty.

The UA-CVAE framework architecture, showing the integration of the Prior and Recognition networks with a GPT-2 backbone via a Combination Network.

Two variants of the combination network were tested:

  • UA-CVAE(M): Uses standard linear layers.
  • UA-CVAE(C): Uses a 1D convolution layer to merge the signals.

Utterance Entailment (UE) Score

To quantify contextual coherence, the authors introduced the UE score. By fine-tuning a BERT model on the SNLI (Natural Language Inference) corpus, the system evaluates the logical relationship between a dialogue context (premise) and a generated response (hypothesis). Responses are scored based on entailment (+1), neutral (0), or contradiction (-1), providing a metric that correlates more closely with human judgment of coherence than traditional overlap metrics.

Experimental Setup

The framework was benchmarked on two key datasets:

  • ConvAI2: Focused on personalized dialogue based on predefined persona profiles.
  • EMPATHETICDIALOGS: Focused on generating emotionally appropriate responses to personal situations.

Results and Discussion

The UA-CVAE models achieved substantial gains in coherence without sacrificing diversity or fluency. As shown in the experimental results, the UE-score improved significantly on the ConvAI2 dataset compared to standard CVAE baselines.

Automatic evaluation results across ConvAI2 and Empathetic Dialogues datasets, highlighting the superior UE-scores of the UA-CVAE variants.

Key Insights

  • Variance as a Signal: Treating latent variance as a measure of uncertainty allows the model to dynamically adjust its reliance on context versus the latent variable, preventing "hallucinations" when the context is ambiguous.
  • NLI for Coherence: The UE-score proved to be a strong proxy for human judgment, offering a more meaningful evaluation of dialogue logic than BLEU or ROUGE.
  • Performance: The 1D convolution variant (UA-CVAE(C)) slightly outperformed the linear version, suggesting it is more effective at merging uncertainty signals.

Conclusion

The UA-CVAE framework demonstrates that explicitly modeling uncertainty can significantly enhance the logical consistency of diverse dialogue agents. While the current work focuses on aleatoric uncertainty, future research may explore epistemic uncertainty to further refine model performance. The introduction of the UE-score also provides a valuable tool for the automated evaluation of conversational coherence.

Summary updated

Abstract

In recent years, latent variable models, such as the Conditional Variational Auto Encoder (CVAE), have been applied to both personalized and empathetic dialogue generation. Prior work have largely focused on generating diverse dialogue responses that exhibit persona consistency and empathy. However, when it comes to the contextual coherence of the generated responses, there is still room for improvement. Hence, to improve the contextual coherence, we propose a novel Uncertainty Aware CVAE (UA-CVAE) framework. The UA-CVAE framework involves approximating and incorporating the aleatoric uncertainty during response generation. We apply our framework to both personalized and empathetic dialogue generation. Empirical results show that our framework significantly improves the contextual coherence of the generated response. Additionally, we introduce a novel automatic metric for measuring contextual coherence, which was found to correlate positively with human judgement.
BibTeX
@inproceedings{lee2022improving,
  author = {Jing Yang Lee and Kong Aik Lee and Woon-Seng Gan},
  title = {{Improving Contextual Coherence in Variational Personalized and Empathetic Dialogue Agents}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=Xq8_abyPFgU},
}
Figure