Abstract
Traditional AI agents for visual guessing games often rely on human-provided category labels, limiting their ability to generalize to novel objects. We present an "imagination" module that enables agents to learn perceptual representations directly from pixels using Regularized Auto-Encoders and a context-aware triplet loss. This approach yields an 8.26% improvement in zero-shot accuracy on the CompGuessWhat?! benchmark and significantly enhances performance when labels are unavailable at inference time.
Introduction
In visual grounding tasks like GuessWhat?!, a "Guesser" agent must identify a target object by asking an "Oracle" yes/no questions. Current state-of-the-art models suffer from a "cheating" problem: they rely on "gold category labels" (e.g., "donut") during both training and testing. This reliance creates an unrealistic advantage and leads to failure when the model encounters unseen object categories. To address this, we propose a move toward "perceptual simulation"—the human ability to aggregate multi-modal memories to understand a concept, even if the specific label is missing or new.
The Imagination Module
We integrate a novel Imagination Module into the Grounded Dialogue State Encoder (GDSE) framework. This module uses Regularized Auto-Encoders (RAEs) to learn informative latent embeddings ($z$) from object image crops. Unlike standard Variational Auto-Encoders, which can suffer from posterior collapse, RAEs offer greater stability and better learning of informative latent spaces.
The core of the module is a Context-Aware Triplet Loss ($L_{REC}^{IMG}$). This loss function ensures that the learned representations are both category-aware (clustering similar visual features) and context-aware (distinguishing the target from other objects in the same scene). By comparing an object to its neighbors in the same image, the model learns which visual features are actually discriminative in a specific situation. Crucially, at inference time, the encoder generates these "imagined" embeddings ($z$) directly from pixels, removing the need for external labels.
Evaluation and Results
The module was evaluated on GuessWhat?! and CompGuessWhat?!, the latter specifically designed for zero-shot generalization.
Zero-Shot Generalization
The imagination-based model demonstrated superior performance in identifying objects from categories never seen during training. It achieved an 8.26% improvement in gameplay accuracy over previous state-of-the-art models. Qualitative success was noted in cases where the model correctly identified novel objects, such as an "antelope," by treating it as a member of a broader "animal" super-category.
Robustness and Attribute Reasoning
When gold labels were withheld at test time, the imagination module improved Guesser accuracy by a massive 12.86%. Furthermore, the Oracle's ability to answer questions about physical attributes like color and shape improved by 5.81% and 1.34%, respectively. This indicates that the latent space successfully captures meaningful physical characteristics that labels alone might miss.
Conclusion
By replacing rigid category labels with learned visual "imaginations," AI agents can better navigate the "long tail" of rare or novel objects. While multi-task training presents some challenges in balancing question generation and answering, the significant gains in zero-shot scenarios highlight the potential of perceptual simulation for more robust and human-like visual grounding.
Summary updated