Multi-Dimension Attention for Multi-Turn Dialog Generation (Student Abstract)

Billal Belainine, Fatiha Sadat, Mounir BoukadoumAAAI 2022 · DOI

Abstract

This paper introduces the Multi-Dimension Attention (MDA) model, designed to enhance multi-turn dialog generation. By implementing a dual-axis position coding scheme and replacing standard dot-product attention with cosine similarity, the MDA model achieves superior response accuracy and reduced memory consumption compared to traditional Transformer and RNN-based architectures.

Introduction

Open-domain, multi-turn dialog systems require maintaining a coherent conversation history to generate relevant responses. Traditional Recurrent Neural Networks (RNNs), such as the Hierarchical Encoder-Decoder (HRED), often struggle with long-term dependencies, leading to a "forgetting" problem where earlier context is lost. While the Transformer architecture addresses some of these issues, it typically treats conversation history as a flat sequence. This approach fails to capture the inherent hierarchy of dialog: words within utterances and utterances within a chronological sequence. The MDA model addresses this by explicitly modeling the structural dependencies of multi-turn conversations.

Methodology

The MDA model incorporates two primary innovations into the Transformer architecture to better handle hierarchical context:

  1. Double Position Coding: To preserve the hierarchical structure, the model utilizes two vectors for each input embedding. Vertical Coding represents the position of a word within an individual utterance, while Horizontal Coding represents the position of the utterance within the overall dialog history.
  2. Multi-dimension Attention (MDA):
    • Cosine Similarity: The model replaces standard dot-product attention with cosine similarity. This eliminates the need for scaling factors and results in a lower memory footprint and more stable learning.
  3. 3D Attention Stack: The decoder computes attention across a 3D stack of matrices representing the entire dialog history, allowing the model to capture dependencies between the output and every word in the history simultaneously.
The MDA architecture featuring double position coding and a 3D multi-dimension attention stack.

Experimental Setup and Results

The model was evaluated using the DailyDialog corpus, a standard benchmark consisting of high-quality human-to-human conversations. Performance was measured against three baselines: a standard Transformer, HRED, and ReCoSa (Relevant Context with Self-Attention).

Comparative performance metrics for MDA and baseline models on the DailyDialog corpus.

As shown in the results table, the MDA model consistently outperformed all baseline models across multiple metrics. MDA achieved a Perplexity (PPL) of 79.62, outperforming the Transformer (83.41) and HRED (85.08). Notably, the BLEU score for MDA reached 2.87, more than doubling the performance of ReCoSa (1.72) and significantly exceeding the basic Transformer (0.42). Furthermore, MDA achieved the highest scores in semantic similarity metrics, including Average (0.671), Greedy (0.524), and Extrema (0.398).

Discussion and Conclusion

The success of the MDA model highlights the importance of structural awareness in dialog modeling. By baking word-level and utterance-level positions into the input, the model avoids the need to learn these structures from scratch. Additionally, the use of cosine similarity provides a clearer "audit trail," allowing researchers to track which specific words or turns influenced a generated response, thereby improving model explainability.

While the automated metrics are strong, future work should involve scaling the model to larger, noisier datasets and conducting qualitative human evaluations to further validate the naturalness and coherence of the generated responses.

Summary updated

BibTeX
@inproceedings{belainine2022multi,
  author = {Billal Belainine and Fatiha Sadat and Mounir Boukadoum},
  title = {{Multi-Dimension Attention for Multi-Turn Dialog Generation (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21591},
}
Figure