Improving Context Modelling in Multimodal Dialogue Generation

Shubham Agarwal, Ondrej Dusek, Ioannis Konstas, Verena Rieser

Abstract

In this work, we investigate the task of textual response generation in a multimodal task-oriented dialogue system. Our work is based on the recently released Multimodal Dialogue (MMD) dataset (Saha et al., 2017) in the fashion domain. We introduce a multimodal extension to the Hierarchical Recurrent Encoder-Decoder (HRED) model and show that this extension outperforms strong baselines in terms of text-based similarity metrics. We also showcase the shortcomings of current vision and language models by performing an error analysis on our system's output.

Introduction: The Challenge of Multimodal Context

Most conversational AI systems are "unimodal," meaning they only process text. However, in real-world scenarios like online shopping, dialogue is inherently "multimodal"—users ask questions about images, and agents respond with galleries of products.

The core challenge addressed here is context modelling. In a complex conversation, a user might say, "Show me something similar to the 4th image but in a different material." To answer correctly, the system must remember not just the text of the previous turn, but the specific set of images shown.

A sample dialogue from the MMD dataset demonstrating the complexity of multimodal interactions in fashion retail.

Previous approaches often "unrolled" these images, treating each one as a separate dialogue turn. This paper argues that this approach loses the structural relationship between images and text, making it harder for the model to reason over long-term context.

Methodology: Aggregated Hierarchical Encoding

The authors build upon the Hierarchical Recurrent Encoder-Decoder (HRED) framework. While a standard HRED uses two levels of RNNs (one for words in a sentence and one for sentences in a dialogue), this version adds a third dimension: Vision.

Multimodal Integration and Aggregation

Instead of processing images one by one, the model takes all images from a single turn, passes them through a pre-trained VGG-19 network to get feature vectors, and then uses a linear layer to create an aggregated "global" image representation for that turn. This allows the model to maintain the natural rhythm of the conversation, making it easier to resolve references like "the 4th image."

Comparison of context representation: the original 'unrolled' approach vs. the proposed aggregated unit approach.

Attention Mechanism

The text representation and the aggregated image representation are concatenated and fed into a "Context RNN." The decoder uses Luong-style attention, allowing the model to focus on specific parts of the previous dialogue context while generating a response. The model is trained end-to-end using cross-entropy loss on next-word prediction.

The Multimodal Dialogue (MMD) Dataset

The study utilizes the MMD dataset, a large-scale corpus released in 2017 focused on the fashion domain, comprising over 120,000 chat sessions and 3.5 million context-response pairs. It includes sophisticated user queries involving quantitative inference (e.g., "show more of the 3rd product"), domain knowledge, and co-reference resolution. A key contribution of this work is re-processing the raw logs to ensure that multiple images in a single turn are treated as a single unit of context, rather than being unrolled into separate turns.

Key Results and Insights

The proposed M-HRED-attn model outperformed the previous state-of-the-art baseline across all primary text-based metrics:

  • BLEU-4: Achieved 0.4451, a significant jump from the baseline's 0.3767.
  • METEOR: Improved to 0.3371 (from 0.2847).
  • ROUGE-L: Improved to 0.6799 (from 0.6235).

Key Findings

  • Context Length Matters: Increasing the context window from 2 turns to 5 turns significantly improved response quality, confirming that "memory" is vital for multimodal tasks.
  • The Power of Attention: Adding an attention mechanism provided a larger performance boost than simply adding multimodal inputs, suggesting that how the model retrieves history is as important as what it sees.
  • The Vision Bottleneck: Despite improvements, the model still struggles with fine-grained visual details (e.g., distinguishing a "casual fit" from a "woven" texture). This indicates that the bottleneck may lie in visual feature extraction rather than dialogue logic.

Conclusion and Future Work

While the model is better at handling the structure of multimodal dialogue, it occasionally ignores visual nuances in favor of textual patterns. Future directions include utilizing more modern vision architectures like ResNet, fine-tuning the image encoder and dialogue model together (end-to-end), and developing methods to better capture specific attributes like material and fit that are currently smoothed over by global representations.

Summary updated

Abstract

In this work, we investigate the task of textual response generation in a multimodal task-oriented dialogue system. Our work is based on the recently released Multimodal Dialogue (MMD) dataset (Saha et al., 2017) in the fashion domain. We introduce a multimodal extension to the Hierarchical Recurrent Encoder-Decoder (HRED) model and show that this extension outperforms strong baselines in terms of text-based similarity metrics. We also showcase the shortcomings of current vision and language models by performing an error analysis on our system's output.
BibTeX
@inproceedings{agarwal2018improving,
  author = {Shubham Agarwal and Ondrej Dusek and Ioannis Konstas and Verena Rieser},
  title = {{Improving Context Modelling in Multimodal Dialogue Generation}},
  booktitle = {INLG},
  year = {2018},
  doi = {10.18653/v1/w18-6514},
}
Figure