Abstract
Generating coherent and informative responses in multi-turn dialogue systems remains a significant challenge. This paper introduces the Conversational Semantic Relationship RNN (CSRR), which employs a three-tier hierarchy of latent variables to explicitly model semantic dependencies. By distinguishing between global discourse context, shared pair-level topics, and unique utterance-level content, CSRR achieves superior performance in generating diverse and contextually relevant responses.
Introduction
A high-quality response in a multi-turn conversation must balance three critical aspects: global coherence with the conversation history, local relevance to the immediate query, and informational diversity to avoid repetitive or generic outputs. Traditional models like the Hierarchical Recurrent Encoder-Decoder (HRED) often produce "safe" but uninformative responses (e.g., "I don't know") because they lack an explicit mechanism to model the underlying semantic relationships between turns. While Variational Autoencoders (VAEs) have been applied to dialogue, they typically conflate the overarching context with the specific relationship between a query and its response. CSRR addresses this by introducing a structured latent space that separates these dependencies.
Methods
The CSRR model utilizes a Variational Autoencoder framework with a hierarchical latent structure designed to capture different levels of semantic information:
- Discourse-level ($z^c$): Represents the global background and context of the entire conversation history.
- Pair-level ($z^p$): Captures the "consistent semantics" or shared topic between a specific query and its corresponding response.
- Utterance-level ($z^q$ and $z^r$): Models the distinct content of the query and response, respectively, ensuring that the response adds new information rather than just repeating the query.
The architecture employs a Bidirectional GRU for encoding utterances and a context RNN to track the discourse state. A reconstruction loss is applied to ensure the latent variables retain the semantic essence of the input, while KL divergence regularizes the latent space to prevent posterior collapse.
Results
The model was evaluated on the Ubuntu Dialog Corpus (technical support) and the Cornell Movie Dialog Corpus (fictional dialogue). CSRR was compared against several baselines, including HRED, VHRED (Variational HRED), and VHCR (Variational Hierarchical Conversation RNN).
Quantitative analysis shows that CSRR consistently outperforms baselines across embedding similarity metrics (Average, Extrema, Greedy) and diversity metrics (Dist-1, Dist-2). Furthermore, human evaluation confirms that CSRR produces responses that are more coherent, fluent, and informative. For instance, on the Ubuntu dataset, CSRR achieved a coherence score of 3.39 compared to HRED's 2.96.
Conclusion
The CSRR model demonstrates that explicitly modeling the hierarchical semantic relationships in dialogue leads to more human-like and engaging responses. By separating topic from content, the model avoids the trap of word-to-word matching and instead focuses on semantic consistency. Future work includes extending this hierarchy to model deeper conversation histories and incorporating reinforcement learning to better manage topic transitions.
Summary updated