Learning Better Visual Dialog Agents With Pretrained Visual-Linguistic Representation

Tao Tu, Qing Ping, Govindarajan Thattai, Gökhan Tür, Prem Natarajan

Abstract

GuessWhat?! is a visual dialog guessing game which incorporates a Questioner agent that generates a sequence of questions, while an Oracle agent answers the respective questions about a target object in an image. Based on this dialog history between the Questioner and the Oracle, a Guesser agent makes a final guess of the target object. While previous work has focused on dialogue policy optimization and visual-linguistic information fusion, most work learns the vision-linguistic encoding for the three agents solely on the GuessWhat?! dataset without shared and prior knowledge of vision-linguistic representation. To bridge these gaps, this paper proposes new Oracle, Guesser and Questioner models that take advantage of a pretrained vision-linguistic model, VilBert. For Oracle model, we introduce a two-way background/target fusion mechanism to understand both intra and inter-object questions. For Guesser model, we introduce a state-estimator that best utilizes Vilbert's strength in single-turn referring expression comprehension. For the Questioner, we share the state-estimator from pretrained Guesser with Questioner to guide the question generator. Experimental results show that our proposed models outperform state-of-the-art models significantly by 7%, 10%, 12% for Oracle, Guesser and End-to-End Questioner respectively.

Introduction

Visual Dialog tasks, such as the "GuessWhat?!" game, require agents to reason over both visual and linguistic information in a multi-turn conversational setting. The game involves three distinct roles: a Questioner that asks binary (Yes/No) questions to locate a secret object, an Oracle that answers these questions based on ground truth, and a Guesser that identifies the target object after the dialog concludes.

Example of the GuessWhat?! visual dialog game where agents must identify a target object through a series of questions.

Historically, these agents were trained from scratch on specific datasets, leading to a "knowledge gap" where models struggled with complex visual concepts like color, shape, or spatial relations. Furthermore, previous models often treated the entire dialog as one long string of text, failing to track how the "belief" about which object is the target changes turn-by-turn.

Methodology

The researchers propose a unified framework leveraging VilBERT, a transformer-based model pretrained on millions of image-caption pairs, to provide foundational visual-linguistic "common sense."

VilBERT-Oracle

The Oracle model incorporates a "two-way background/target fusion" mechanism. By fusing features from both the specific target object and the overall image background with the question, the model can accurately answer "inter-object" questions (e.g., "Is it to the left of the person in red?").

VilBERT-Guesser and the State-Estimator

A key innovation is the introduction of a state-estimator for the Guesser. Instead of processing the entire dialog at once, the model updates a probability distribution (or "belief") across all candidate objects after each question-answer turn. This approach leverages VilBERT’s proficiency in single-turn referring expression comprehension to eliminate candidates iteratively.

The VilBERT-Guesser architecture utilizes a state-estimator to update object probabilities turn-by-turn based on dialog history.

VilBERT-Questioner

The Questioner agent is enhanced by sharing the state-estimator from the pretrained Guesser. By monitoring the Guesser's current belief state, the Questioner can generate more strategic questions designed to maximize information gain and narrow down the search space effectively.

Experimental Results

The models were evaluated on the GuessWhat?! dataset, which includes 155k dialogues and 66k unique images. The proposed VilBERT-based agents achieved significant improvements over previous state-of-the-art (SOTA) methods:

  • Oracle Accuracy: Increased by 7%, reaching 85.0%.
  • Guesser Accuracy: Increased by 10%, reaching 76.5%.
  • End-to-End Success Rate: The overall game success rate improved by 12%, reaching 62.8%.

Beyond accuracy, the VilBERT-Questioner demonstrated higher semantic diversity and was significantly less likely to repeat questions compared to baseline models.

Conclusion

By integrating pretrained visual-linguistic representations and implementing turn-level state tracking, this work demonstrates that "common sense" knowledge and incremental reasoning are vital for complex visual dialog tasks. While this study focused on supervised learning, future work may explore Reinforcement Learning to further optimize the Questioner's goal-oriented strategies and address the computational complexity of transformer-based multi-agent systems.

Summary updated

Abstract

GuessWhat?! is a visual dialog guessing game which incorporates a Questioner agent that generates a sequence of questions, while an Oracle agent answers the respective questions about a target object in an image. Based on this dialog history between the Questioner and the Oracle, a Guesser agent makes a final guess of the target object. While previous work has focused on dialogue policy optimization and visual-linguistic information fusion, most work learns the vision-linguistic encoding for the three agents solely on the GuessWhat?! dataset without shared and prior knowledge of vision-linguistic representation. To bridge these gaps, this paper proposes new Oracle, Guesser and Questioner models that take advantage of a pretrained vision-linguistic model, VilBert. For Oracle model, we introduce a two-way background/target fusion mechanism to understand both intra and inter-object questions. For Guesser model, we introduce a state-estimator that best utilizes Vilbert's strength in single-turn referring expression comprehension. For the Questioner, we share the state-estimator from pretrained Guesser with Questioner to guide the question generator. Experimental results show that our proposed models outperform state-of-the-art models significantly by 7%, 10%, 12% for Oracle, Guesser and End-to-End Questioner respectively.
BibTeX
@inproceedings{tu2021learning,
  author = {Tao Tu and Qing Ping and Govindarajan Thattai and Gökhan Tür and Prem Natarajan},
  title = {{Learning Better Visual Dialog Agents With Pretrained Visual-Linguistic Representation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=SUlOJaZ3Pn},
}
Figure