Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog

Zhe Gan, Yu Cheng, Ahmed El Kholy, Linjie Li, Jingjing Liu, Jianfeng Gao

Abstract

This paper presents a new model for visual dialog, Recurrent Dual Attention Network (ReDAN), using multi-step reasoning to answer a series of questions about an image. In each question-answering turn of a dialog, ReDAN infers the answer progressively through multiple reasoning steps. In each step of the reasoning process, the semantic representation of the question is updated based on the image and the previous dialog history, and the recurrently-refined representation is used for further reasoning in the subsequent step. On the VisDial v1.0 dataset, the proposed ReDAN model achieves a new state-of-the-art of 64.47% NDCG score. Visualization on the reasoning process further demonstrates that ReDAN can locate context-relevant visual and textual clues via iterative refinement, which can lead to the correct answer step-by-step.

Introduction

Visual Dialog is a complex task that extends standard Visual Question Answering (VQA) into a conversational setting. It requires a model to maintain a coherent dialogue about an image, necessitating a deep understanding of visual context, textual history, and coreference resolution. For instance, to answer "Is he wearing a hat?", a model must identify the referent of "he" from previous turns and then locate that specific individual in the image.

Traditional models often employ "single-shot" attention mechanisms, which process the image and history in a single pass. However, human reasoning is often iterative; we frequently re-examine evidence as our understanding of a query evolves. To mimic this, we introduce the Recurrent Dual Attention Network (ReDAN), which employs multi-step reasoning to progressively refine its focus on relevant visual and textual information.

Methodology

The ReDAN architecture is designed to iteratively update its internal representation of a question by attending to both visual and textual memories.

Memory Generation

The model first constructs two types of memory:

  • Visual Memory: Features are extracted from 36 object regions in the image using a pre-trained Faster R-CNN.
  • Textual Memory: A BiLSTM encodes the image caption and the sequence of previous question-answer pairs into a searchable textual history.

Multi-step Reasoning

The core of ReDAN is its recurrent reasoning loop. In each step $t \in \{1, \dots, T\}$, the model uses its current question state to perform Dual Attention:

  1. Visual Attention: It identifies relevant regions in the visual memory.
  2. Textual Attention: It identifies relevant snippets in the dialog history.

The question representation is then updated based on these attended features. As shown in the figure below, this iterative process allows the model to "sharpen" its attention, moving from broad regions to specific objects or text segments necessary for the final answer.

Overview of the ReDAN framework and its multi-step reasoning process.

Answer Decoding

ReDAN supports both discriminative and generative decoders. A discriminative decoder ranks a set of 100 candidate answers, while a generative decoder synthesizes an answer word-by-word. We also utilize a rank aggregation method to combine the strengths of both approaches, significantly improving overall performance.

Experimental Results

We evaluated ReDAN on the VisDial v1.0 dataset, which contains approximately 1.2 million question-answer pairs across 123k images. The primary metric is the Normalized Discounted Cumulative Gain (NDCG).

Performance

ReDAN achieved a state-of-the-art NDCG of 64.47% on the test-std set, a substantial improvement over previous models like DAN (59.36%). Our ablation studies confirmed that increasing the number of reasoning steps ($T$) from 1 to 3 consistently improves performance, validating the multi-step approach.

Qualitative Analysis

Visualization of the reasoning process reveals that the model successfully performs coreference resolution. In the first step, attention might be distributed across the entire scene. By the second step, the model focuses on the specific object mentioned in the history (e.g., the boy's shorts), leading to a correct "yes" answer.

Conclusion

ReDAN demonstrates the power of multi-step reasoning for Visual Dialog. By recurrently refining its attention over visual and textual memories, the model achieves superior performance and provides interpretable reasoning paths. Future work will address remaining challenges such as accurate counting in complex scenes and optimizing computational efficiency through adaptive reasoning steps.

Summary updated

Abstract

This paper presents a new model for visual dialog, Recurrent Dual Attention Network (ReDAN), using multi-step reasoning to answer a series of questions about an image. In each question-answering turn of a dialog, ReDAN infers the answer progressively through multiple reasoning steps. In each step of the reasoning process, the semantic representation of the question is updated based on the image and the previous dialog history, and the recurrently-refined representation is used for further reasoning in the subsequent step. On the VisDial v1.0 dataset, the proposed ReDAN model achieves a new state-of-the-art of 64.47% NDCG score. Visualization on the reasoning process further demonstrates that ReDAN can locate context-relevant visual and textual clues via iterative refinement, which can lead to the correct answer step-by-step.
BibTeX
@inproceedings{gan2019multi,
  author = {Zhe Gan and Yu Cheng and Ahmed El Kholy and Linjie Li and Jingjing Liu and Jianfeng Gao},
  title = {{Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=RZU6JVJs5X3},
}
Figure