Modeling Semantic Relationship in Multi-turn Conversations with Hierarchical Latent Variables

Lei Shen, Yang Feng, Haolan Zhan

Abstract

Multi-turn conversations consist of complex semantic structures, and it is still a challenge to generate coherent and diverse responses given previous utterances. It's practical that a conversation takes place under a background, meanwhile, the query and response are usually most related and they are consistent in topic but also different in content. However, little work focuses on such hierarchical relationship among utterances. To address this problem, we propose a Conversational Semantic Relationship RNN (CSRR) model to construct the dependency explicitly. The model contains latent variables in three hierarchies. The discourse-level one captures the global background, the pair-level one stands for the common topic information between query and response, and the utterance-level ones try to represent differences in content. Experimental results show that our model significantly improves the quality of responses in terms of fluency, coherence and diversity compared to baseline methods.

Introduction

Generating human-like responses in multi-turn conversations is a significant challenge in natural language processing. In real-world dialogues, a query and its response are not isolated strings; they are bound by a shared global background and a specific topic of discussion, while simultaneously containing unique information. Existing models, such as HRED and VHRED, often fail to capture these nuanced dependencies, frequently resulting in generic or logically disconnected responses—the well-known "I don't know" problem.

The authors argue that effective response generation requires explicit representation of the hierarchical semantic relationships between the context, the query, and the response. By separating topic from content, a model can maintain coherence while introducing informative diversity.

Methods: The CSRR Model

The proposed Conversational Semantic Relationship RNN (CSRR) utilizes Variational Autoencoders (VAEs) organized into three distinct hierarchies to explicitly model dialogue dependencies:

  1. Discourse-level ($z^c$): Captures the global background of the entire conversation history.
  2. Pair-level ($z^p$): Represents the shared topic information specifically between the query and the intended response.
  3. Utterance-level ($z^q$ and $z^r$): Models the specific content differences—the "query-specific" and "response-specific" details that distinguish the two utterances.
The architecture of the CSRR model, illustrating the hierarchical latent variables $z^c, z^p, z^q,$ and $z^r$ and their relationships with utterances $u$.

The model employs a Bidirectional GRU to encode utterances and a context RNN to track dialogue flow. Training involves optimizing a variational lower-bound (ELBO) that includes reconstruction loss to preserve semantic meaning and KL divergence terms for latent space regularization.

Experimental Evaluation

The CSRR model was evaluated on two benchmarks: the Ubuntu Dialog Corpus (technical support logs) and the Cornell Movie Dialog Corpus (fictional scripts).

Quantitative Results

CSRR significantly outperformed state-of-the-art baselines, including HRED, VHRED, and VHCR, across both automatic and human metrics. As shown in the table below, CSRR achieved superior scores in embedding-based similarity (Average, Extrema, Greedy) and diversity metrics (Dist-1, Dist-2).

Comparison of CSRR against baseline models on the Ubuntu and Cornell Movie Dialog datasets across multiple metrics.

Qualitative and Human Assessment

In blind tests with human annotators, CSRR was rated highest for Coherence, Fluency, and Informativeness. Qualitative case studies demonstrated that while baselines often defaulted to generic answers, CSRR provided specific, context-aware instructions, such as identifying the need for specific hardware drivers in technical queries.

Conclusion

The CSRR model demonstrates the power of explicit hierarchical modeling in dialogue systems. By separating discourse, topic, and content, it achieves more coherent and informative multi-turn conversations. Future work may extend this pair-level modeling across all previous turns and incorporate Reinforcement Learning to better manage topic transitions in extended dialogues.

Summary updated

Abstract

Multi-turn conversations consist of complex semantic structures, and it is still a challenge to generate coherent and diverse responses given previous utterances. It's practical that a conversation takes place under a background, meanwhile, the query and response are usually most related and they are consistent in topic but also different in content. However, little work focuses on such hierarchical relationship among utterances. To address this problem, we propose a Conversational Semantic Relationship RNN (CSRR) model to construct the dependency explicitly. The model contains latent variables in three hierarchies. The discourse-level one captures the global background, the pair-level one stands for the common topic information between query and response, and the utterance-level ones try to represent differences in content. Experimental results show that our model significantly improves the quality of responses in terms of fluency, coherence and diversity compared to baseline methods.
BibTeX
@inproceedings{shen2019modeling,
  author = {Lei Shen and Yang Feng and Haolan Zhan},
  title = {{Modeling Semantic Relationship in Multi-turn Conversations with Hierarchical Latent Variables}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=hdG96n1UGBo},
}
Figure