Abstract
GuessWhat?! is a visual dialog guessing game which incorporates a Questioner agent that generates a sequence of questions, while an Oracle agent answers the respective questions about a target object in an image. Based on this dialog history between the Questioner and the Oracle, a Guesser agent makes a final guess of the target object. While previous work has focused on dialogue policy optimization and visual-linguistic information fusion, most work learns the vision-linguistic encoding for the three agents solely on the GuessWhat?! dataset without shared and prior knowledge of vision-linguistic representation. To bridge these gaps, this paper proposes new Oracle, Guesser and Questioner models that take advantage of a pretrained vision-linguistic model, VilBert. For Oracle model, we introduce a two-way background/target fusion mechanism to understand both intra and inter-object questions. For Guesser model, we introduce a state-estimator that best utilizes Vilbert's strength in single-turn referring expression comprehension. For the Questioner, we share the state-estimator from pretrained Guesser with Questioner to guide the question generator. Experimental results show that our proposed models outperform state-of-the-art models significantly by 7%, 10%, 12% for Oracle, Guesser and End-to-End Questioner respectively.
Introduction
Visual Dialog tasks, such as the "GuessWhat?!" game, require agents to reason over both visual and linguistic information in a multi-turn conversational setting. The game involves three distinct roles: a Questioner that asks binary (Yes/No) questions to locate a secret object, an Oracle that answers these questions based on ground truth, and a Guesser that identifies the target object after the dialog concludes.
Historically, these agents were trained from scratch on specific datasets, leading to a "knowledge gap" where models struggled with complex visual concepts like color, shape, or spatial relations. Furthermore, previous models often treated the entire dialog as one long string of text, failing to track how the "belief" about which object is the target changes turn-by-turn.
Methodology
The researchers propose a unified framework leveraging VilBERT, a transformer-based model pretrained on millions of image-caption pairs, to provide foundational visual-linguistic "common sense."
VilBERT-Oracle
The Oracle model incorporates a "two-way background/target fusion" mechanism. By fusing features from both the specific target object and the overall image background with the question, the model can accurately answer "inter-object" questions (e.g., "Is it to the left of the person in red?").
VilBERT-Guesser and the State-Estimator
A key innovation is the introduction of a state-estimator for the Guesser. Instead of processing the entire dialog at once, the model updates a probability distribution (or "belief") across all candidate objects after each question-answer turn. This approach leverages VilBERT’s proficiency in single-turn referring expression comprehension to eliminate candidates iteratively.
VilBERT-Questioner
The Questioner agent is enhanced by sharing the state-estimator from the pretrained Guesser. By monitoring the Guesser's current belief state, the Questioner can generate more strategic questions designed to maximize information gain and narrow down the search space effectively.
Experimental Results
The models were evaluated on the GuessWhat?! dataset, which includes 155k dialogues and 66k unique images. The proposed VilBERT-based agents achieved significant improvements over previous state-of-the-art (SOTA) methods:
- Oracle Accuracy: Increased by 7%, reaching 85.0%.
- Guesser Accuracy: Increased by 10%, reaching 76.5%.
- End-to-End Success Rate: The overall game success rate improved by 12%, reaching 62.8%.
Beyond accuracy, the VilBERT-Questioner demonstrated higher semantic diversity and was significantly less likely to repeat questions compared to baseline models.
Conclusion
By integrating pretrained visual-linguistic representations and implementing turn-level state tracking, this work demonstrates that "common sense" knowledge and incremental reasoning are vital for complex visual dialog tasks. While this study focused on supervised learning, future work may explore Reinforcement Learning to further optimize the Questioner's goal-oriented strategies and address the computational complexity of transformer-based multi-agent systems.
Summary updated