Abstract
Spatial commonsense—the knowledge about spatial position and relationship between objects—is a fundamental component of human intelligence that remains elusive for many AI systems. While pretrained language models (PLMs) excel in various NLP tasks, they often struggle with spatial reasoning due to reporting bias in text. This paper investigates whether visual signals, specifically from image synthesis models (ISMs), provide a more robust foundation for spatial commonsense. By introducing a benchmark focused on object scales and positional relationships, we demonstrate that ISMs are more capable of learning accurate and consistent spatial knowledge than text-only or vision-language models. Furthermore, we show that this visually-derived knowledge can significantly enhance performance in natural language understanding tasks.
Introduction: The Reporting Bias in Text
Humans possess an intuitive grasp of spatial relationships: we know a lion is larger than a girl and that a cyclist sits atop a bicycle. However, text-only models like BERT or RoBERTa frequently fail at these basic comparisons. This is largely due to reporting bias: people rarely write down the obvious. While a corpus might contain thousands of instances of "the lion roared," it seldom explicitly states "the lion is larger than the girl."
Because images must adhere to physical laws and scale to appear realistic, they serve as a superior repository for this "unspoken" spatial knowledge. This research explores whether generative models can bridge the knowledge gap left by text-only training.
Methodology
The researchers compared three types of models to evaluate their spatial reasoning capabilities:
- Text-only PLMs: BERT and RoBERTa.
- Vision-Language Pretrained Models (VL-PTMs): Models like VinVL, trained on aligned text-image pairs.
- Image Synthesis Models (ISMs): Generative models (e.g., VQGAN+CLIP) that create images from text prompts.
To probe these models, the authors used comparative prompts. For PLMs, they utilized masked language modeling (e.g., "A sofa is [MASK] than a mountain"). For ISMs, the models were prompted to generate a scene (e.g., "A sofa and a mountain"). The resulting images were then analyzed using automated object detectors (ISM-Box) and human evaluation (ISM-Human) to determine if the relative scales were correct.
Key Results and Insights
Superior Accuracy and Consistency
The study found that ISMs significantly outperform text-based models in spatial accuracy. In size comparison tasks, human evaluation of ISM outputs reached 72.7% accuracy, compared to 54.1% for RoBERTa and 61.8% for the vision-language model VinVL. The performance gap was even wider in positional reasoning, where ISMs reached 73.4% accuracy while text-only models struggled at approximately 31%.
Furthermore, ISMs demonstrated much higher logical consistency. In symmetry tests (e.g., if a coin is smaller than a tire, then a tire is larger than a coin), ISMs maintained over 80% consistency, whereas PLMs often failed to correctly reverse the relationship, dropping to roughly 40%.
Visual Knowledge for NLU
The researchers proved that this implicit visual knowledge can be integrated back into natural language processing. By using an ISM to generate a "mental image" for a question-answering model (UnifiedQA), they observed measurable improvements in the model's ability to answer spatial commonsense questions across the board.
Conclusion
This research highlights that spatial commonsense is an emergent property of generative visual models. Unlike text models that rely on statistical co-occurrences of words, ISMs must learn a comprehensive internal representation of object scales and positions to reconstruct realistic scenes. While limitations exist—such as the difficulty of automatically parsing generated images—tapping into the visual signals of ISMs offers a promising path for building AI with a more human-like understanding of the physical world.
Summary updated