Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games

Alessandro Suglia, Antonio Vergari, Ioannis Konstas, Yonatan Bisk, Emanuele Bastianelli, Andrea Vanzo, Oliver Lemon

Abstract

Traditional AI agents for visual guessing games often rely on human-provided category labels, limiting their ability to generalize to novel objects. We present an "imagination" module that enables agents to learn perceptual representations directly from pixels using Regularized Auto-Encoders and a context-aware triplet loss. This approach yields an 8.26% improvement in zero-shot accuracy on the CompGuessWhat?! benchmark and significantly enhances performance when labels are unavailable at inference time.

Introduction

In visual grounding tasks like GuessWhat?!, a "Guesser" agent must identify a target object by asking an "Oracle" yes/no questions. Current state-of-the-art models suffer from a "cheating" problem: they rely on "gold category labels" (e.g., "donut") during both training and testing. This reliance creates an unrealistic advantage and leads to failure when the model encounters unseen object categories. To address this, we propose a move toward "perceptual simulation"—the human ability to aggregate multi-modal memories to understand a concept, even if the specific label is missing or new.

The Imagination Module

We integrate a novel Imagination Module into the Grounded Dialogue State Encoder (GDSE) framework. This module uses Regularized Auto-Encoders (RAEs) to learn informative latent embeddings ($z$) from object image crops. Unlike standard Variational Auto-Encoders, which can suffer from posterior collapse, RAEs offer greater stability and better learning of informative latent spaces.

Comparison of category labels (c) and latent embeddings (z) for different object crops, showing how the model maps pixels to a latent space.

The core of the module is a Context-Aware Triplet Loss ($L_{REC}^{IMG}$). This loss function ensures that the learned representations are both category-aware (clustering similar visual features) and context-aware (distinguishing the target from other objects in the same scene). By comparing an object to its neighbors in the same image, the model learns which visual features are actually discriminative in a specific situation. Crucially, at inference time, the encoder generates these "imagined" embeddings ($z$) directly from pixels, removing the need for external labels.

Evaluation and Results

The module was evaluated on GuessWhat?! and CompGuessWhat?!, the latter specifically designed for zero-shot generalization.

Zero-Shot Generalization

The imagination-based model demonstrated superior performance in identifying objects from categories never seen during training. It achieved an 8.26% improvement in gameplay accuracy over previous state-of-the-art models. Qualitative success was noted in cases where the model correctly identified novel objects, such as an "antelope," by treating it as a member of a broader "animal" super-category.

Zero-shot identification of a "pasticciotto," an unseen category, using latent embeddings (z) derived from visual features.

Robustness and Attribute Reasoning

When gold labels were withheld at test time, the imagination module improved Guesser accuracy by a massive 12.86%. Furthermore, the Oracle's ability to answer questions about physical attributes like color and shape improved by 5.81% and 1.34%, respectively. This indicates that the latent space successfully captures meaningful physical characteristics that labels alone might miss.

Conclusion

By replacing rigid category labels with learned visual "imaginations," AI agents can better navigate the "long tail" of rare or novel objects. While multi-task training presents some challenges in balancing question generation and answering, the significant gains in zero-shot scenarios highlight the potential of perceptual simulation for more robust and human-like visual grounding.

Summary updated

BibTeX
@inproceedings{suglia2020imagining,
  author = {Alessandro Suglia and Antonio Vergari and Ioannis Konstas and Yonatan Bisk and Emanuele Bastianelli and Andrea Vanzo and Oliver Lemon},
  title = {{Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=4jTA2kCTSFK},
}
Figure