Abstract
This paper introduces the Recurrent Dual Attention Network (ReDAN), a novel framework for Visual Dialog that utilizes multi-step reasoning to answer a sequence of questions about an image. By iteratively refining its attention over both visual and textual memories, ReDAN achieves a state-of-the-art NDCG score of 64.47% on the VisDial v1.0 dataset. The model demonstrates that a recurrent refinement process allows for more precise localization of context-relevant clues, leading to more accurate answers in complex conversational settings.
Introduction
Visual Dialog is a significant advancement over standard Visual Question Answering (VQA). It requires a model to not only understand an image but also maintain a "memory" of the conversation to resolve pronouns and understand context. For instance, answering "What color is his shirt?" requires identifying the person mentioned in previous turns.
Most existing models employ a single-step attention mechanism, taking a single "glimpse" at the image and history. The authors of ReDAN argue that this is insufficient for complex queries. Human reasoning is often iterative; we revisit specific parts of an image or text once we have a partial understanding of the context. ReDAN mimics this behavior by performing multiple reasoning steps to find answers that may be confined to small image regions or specific snippets of past dialogue.
The ReDAN Framework
The ReDAN model consists of three primary components designed to handle the interplay between visual and textual information:
- Memory Generation Module: This module uses a pre-trained Faster R-CNN to extract object-level visual features and a BiLSTM to encode the dialogue history into a "textual memory."
- Multi-step Reasoning Module: This is the core of the framework. Instead of a single pass, the model performs $T$ reasoning steps. In each step, the question representation is updated by attending to the visual memory and then the textual memory. A Gated Recurrent Unit (GRU) manages these updates, allowing the model to "sharpen" its focus over time.
- Answer Decoding Module: The final multimodal context is fed into either a Discriminative Decoder (ranking candidate answers) or a Generative Decoder (synthesizing answers word-by-word).
Iterative Reasoning and Attention Refinement
A key insight of the ReDAN model is the synergy between its dual attention mechanisms. The image helps the model understand which part of the history is relevant (resolving coreference), while the history helps the model understand which part of the image to zoom in on.
Visualization of the reasoning process reveals that in the first step, the model might have a broad or even incorrect focus. By the second or third step, the recurrent update to the question vector allows the model to "correct" its course. For example, when asked "is he wearing shorts?", the model may initially look at the person's head but eventually narrows its attention to the leg area as it processes the context.
Experimental Results
The model was evaluated on the VisDial v1.0 benchmark, which includes approximately 1.2 million question-answer pairs. ReDAN achieved a state-of-the-art 64.47% NDCG on the test-std server, significantly outperforming previous models like DAN (59.36%).
Experimental results showed that increasing the number of reasoning steps from $T=1$ to $T=3$ consistently improved performance. Furthermore, the authors found that combining generative and discriminative models via rank aggregation provided a significant boost, as generative models performed better on simple "Yes/No" questions while discriminative models excelled at complex queries.
Conclusion and Future Work
ReDAN demonstrates that multi-step reasoning is essential for the nuanced task of Visual Dialog. By iteratively refining attention, the model can locate subtle visual and textual clues that single-step models miss. While the model still struggles with discrete counting tasks (e.g., "How many people?"), its success in localization points toward a promising future for relation-aware encoders that can better understand the semantic and spatial relationships between objects in a scene.
Summary updated