A Speaker-aware Parallel Hierarchical Attentive Encoder-Decoder Model for Multi-turn Dialogue Generation

Zihao Wang, Ming Jiang, Junli WangarXiv 2021 · arXiv

Introduction

In multi-turn conversations, context is paramount. Most current dialogue models treat conversation history as a flat sequence of text, often ignoring the social dynamics and roles of the participants. This lack of "speaker awareness" leads to failure modes where models fail to capture latent properties of a speaker, such as their role or specific knowledge. For instance, a model might generate a redundant response because it does not recognize that the information was already provided by the same speaker. To address this, we present the Speaker-aware Parallel Hierarchical Attentive Encoder-Decoder (PHAED) model, designed to maintain context coherence by explicitly modeling speaker relations.

Methodology

The PHAED architecture introduces three primary innovations to enhance multi-turn dialogue generation:

  1. Speaker-Aware Input Representation: Utterances are prepended with specific tokens—[Speaker-Q] for the initiator and [Speaker-R] for the responder. Aligned turn embeddings are also utilized to track chronological order.
  2. Hierarchical Encoder: This component uses an "Inner-Query Encoding" (standard Transformer block) for intra-utterance processing and a novel "Inter-Query Encoding" with turn-level relative attention. This allows the model to dynamically focus on relevant past turns based on relative position.
  3. Decoder with Turn-Level Recurrence: Inspired by Transformer-XL, the decoder reuses hidden states from previously generated responses. This "memory" mechanism ensures consistency in persona and prevents repetitive or contradictory statements.
The PHAED architecture featuring a hierarchical encoder and a decoder with turn-level recurrence.

Datasets

The model was rigorously evaluated on three multi-turn dialogue benchmarks:

  • DailyDialog: Human-labeled daily life conversations.
  • PersonaChat: Conversations where speakers have assigned personas to test consistency.
  • Ubuntu v2.0: Technical support logs used to evaluate complex, goal-oriented logic.

Results and Analysis

PHAED consistently outperformed state-of-the-art baselines, including HRAN, ReCoSa, and DialoGPT, across both automatic and human metrics.

Automatic Metrics

The model achieved the lowest Perplexity (PPL) and the highest BLEU scores across all datasets. Furthermore, high scores in Distinct-1/2 metrics indicate that PHAED generates more diverse and informative responses compared to generic baselines. Analysis of attention weights revealed that the model often treats the initial query ($Q_1$) as a thematic anchor, consistently referring back to it throughout the dialogue.

Human Evaluation

In blind tests, human annotators rated PHAED significantly higher than DialoGPT in terms of Coherence and Informativeness. The model's ability to reuse its own "thoughts" via turn-level recurrence was cited as a key factor in maintaining long-term dialogue consistency.

Human evaluation results comparing PHAED against state-of-the-art baselines across multiple dimensions.

Conclusion

The PHAED model demonstrates that explicitly modeling speaker identity and incorporating turn-level recurrence significantly improves the coherence and diversity of multi-turn dialogue generation. While currently optimized for dyadic conversations, future work will explore extending this architecture to multi-party interactions and more complex social hierarchies.

Summary updated

Abstract

This paper presents a novel open-domain dialogue generation model emphasizing the differentiation of speakers in multi-turn conversations. Differing from prior work that solely relies on the content of conversation history to generate a response, we argue that capturing relative social relations among utterances (i.e., generated by either the same speaker or different persons) benefits the machine capturing fine-grained context information from a conversation history to improve context coherence in the generated response. Given that, we propose a speaker-aware Parallel Hierarchical Attentive Encoder-Decoder (PHAED) model that aims to model each utterance with the awareness of its speaker and contextual associations with the same speaker's previous messages. Specifically, in a conversation involving two speakers, we regard the utterances from one speaker as responses and those from the other as queries. After understanding queries via our encoder with inner-query and inter-query encodings, our decoder reuses the hidden states of previously generated responses, instead of reconstructing these by the encoder, to generate a new response. Our empirical results show that PHAED outperforms the state-of-the-art in both automatic and human evaluations. Furthermore, our ablation study shows that dialogue models with speaker tokens can generally decrease the possibility of generating non-coherent responses regarding the conversation context.
BibTeX
@misc{wang2021speaker,
  author = {Zihao Wang and Ming Jiang and Junli Wang},
  title = {{A Speaker-aware Parallel Hierarchical Attentive Encoder-Decoder Model for Multi-turn Dialogue Generation}},
  year = {2021},
  eprint = {2110.06823},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.06823},
}
Figure