Answer-Driven Visual State Estimator for Goal-Oriented Visual Dialogue

Zipeng Xu, Fangxiang Feng, Xiaojie Wang, Yushu Yang, Huixing Jiang, Zhongyuan OuyangACM Multimedia 2020 · DOI · OpenReview

TL;DR

This paper introduces the Answer-Driven Visual State Estimator (ADVSE), a novel framework for goal-oriented visual dialogue that explicitly leverages the crucial information provided by answers to guide visual attention and state estimation. By dynamically adjusting focus based on "yes" or "no" responses, ADVSE significantly improves question generation and object guessing performance, achieving state-of-the-art results.

What Problem Does This Paper Address?

Goal-oriented visual dialogue involves agents conversing to achieve a specific objective, like identifying an object in an image. A key challenge is how the "Questioner" agent effectively uses the "Oracle's" answers to refine its understanding of the visual scene and formulate better questions. Existing methods often treat answers as just another piece of text, failing to recognize their critical role in directing attention and updating beliefs. This leads to inefficient questioning and a weak connection between dialogue and visual perception. The paper aims to bridge this gap by developing a system that intelligently uses answers to shape the agent's visual focus.

Methods

The core contribution is the Answer-Driven Visual State Estimator (ADVSE), which comprises two main components:

  • Answer-Driven Focusing Attention (ADFA): This module dynamically updates the agent's visual attention. It first uses a "sharpening operation" to highlight question-relevant regions. Then, based on the answer ("yes" or "no"), it either maintains attention on these regions or shifts it to other areas. This ensures that answers directly influence where the agent looks next.
  • Conditional Visual Information Fusion (CVIF): After ADFA determines the visual attention, CVIF fuses two types of visual information: the overall image context and the differences between the most focused object and others. This fusion is conditioned on the current question-answer pair, allowing the system to prioritize relevant information based on the dialogue state.

These components are integrated with standard encoders (for image and language) and used within Question Generator (QGen) and Guesser models.

Datasets

The paper uses the GuessWhat?! dataset for training and evaluation. This dataset is designed for goal-oriented visual dialogue tasks, specifically object discovery through conversation.

Key Insights and Ideas

  • Answers are not equal: The paper highlights that "yes" and "no" answers have fundamentally different implications for visual attention and belief updating. A "yes" might mean focusing on finer details of a confirmed object, while a "no" suggests exploring entirely new possibilities.
  • Dynamic attention is key: Visual attention shouldn't be static or solely driven by the question. It needs to be fluid and responsive to the dialogue's progression, especially the answers.
  • Answer-driven focus improves efficiency: By explicitly modeling the impact of answers, the system can generate more targeted questions and make more accurate guesses, leading to faster convergence towards the target object.

Key Results

The proposed ADVSE achieved state-of-the-art performance on the GuessWhat?! dataset for both:

  • Question Generation (QGen): Achieved a success rate of 72.73% on "New object" and 69.88% on "New game" in reinforcement learning settings, outperforming previous methods by significant margins (e.g., 9.08 points improvement on "New game" compared to VQG).
  • Guesser: Achieved the lowest test error rate of 33.15%, surpassing existing models.

Qualitative results demonstrate that ADVSE generates highly efficient questions and exhibits reliable visual attention that adapts to the dialogue flow.

Limitations and Future Work

  • Limitations: The paper acknowledges that the joint QGen and Guesser model is combined in a "rather simple way." The ablation studies also show that while ADFA and CVIF are crucial, removing components still leaves a reasonably performing model, suggesting room for further optimization.
  • Future Work: The authors suggest that further exploration into more sophisticated methods for jointly using the ADVSE-QGen and ADVSE-Guesser models is expected. They also hint at the potential for more advanced fusion strategies and attention mechanisms.

Summary updated

Abstract

A goal-oriented visual dialogue involves multi-turn interactions between two agents, Questioner and Oracle. During which, the answer given by Oracle is of great significance, as it provides golden response to what Questioner concerns. Based on the answer, Questioner updates its belief on target visual content and further raises another question. Notably, different answers drive into different visual beliefs and future questions. However, existing methods always indiscriminately encode answers after much longer questions, resulting in a weak utilization of answers. In this paper, we propose an Answer-Driven Visual State Estimator (ADVSE) to impose the effects of different answers on visual states. First, we propose an Answer-Driven Focusing Attention (ADFA) to capture the answer-driven effect on visual attention by sharpening question-related attention and adjusting it by answer-based logical operation at each turn. Then based on the focusing attention, we get the visual state estimation by Conditional Visual Information Fusion (CVIF), where overall information and difference information are fused conditioning on the question-answer state. We evaluate the proposed ADVSE to both question generator and guesser tasks on the large-scale GuessWhat?! dataset and achieve the state-of-the-art performances on both tasks. The qualitative results indicate that the ADVSE boosts the agent to generate highly efficient questions and obtains reliable visual attentions during the reasonable question generation and guess processes.
BibTeX
@inproceedings{xu2020answer,
  author = {Zipeng Xu and Fangxiang Feng and Xiaojie Wang and Yushu Yang and Huixing Jiang and Zhongyuan Ouyang},
  title = {{Answer-Driven Visual State Estimator for Goal-Oriented Visual Dialogue}},
  booktitle = {ACM Multimedia},
  year = {2020},
  doi = {10.1145/3394171.3413668},
}

Loading related content...

Figure