Things not Written in Text: Exploring Spatial Commonsense from Visual Signals

Xiao Liu, Da Yin, Yansong Feng, Dongyan Zhao

Abstract

Spatial commonsense—the knowledge about spatial position and relationship between objects—is a fundamental component of human intelligence that remains elusive for many AI systems. While pretrained language models (PLMs) excel in various NLP tasks, they often struggle with spatial reasoning due to reporting bias in text. This paper investigates whether visual signals, specifically from image synthesis models (ISMs), provide a more robust foundation for spatial commonsense. By introducing a benchmark focused on object scales and positional relationships, we demonstrate that ISMs are more capable of learning accurate and consistent spatial knowledge than text-only or vision-language models. Furthermore, we show that this visually-derived knowledge can significantly enhance performance in natural language understanding tasks.

Introduction: The Reporting Bias in Text

Humans possess an intuitive grasp of spatial relationships: we know a lion is larger than a girl and that a cyclist sits atop a bicycle. However, text-only models like BERT or RoBERTa frequently fail at these basic comparisons. This is largely due to reporting bias: people rarely write down the obvious. While a corpus might contain thousands of instances of "the lion roared," it seldom explicitly states "the lion is larger than the girl."

Spatial commonsense examples: comparing lion size and cyclist positioning through text vs. images.

Because images must adhere to physical laws and scale to appear realistic, they serve as a superior repository for this "unspoken" spatial knowledge. This research explores whether generative models can bridge the knowledge gap left by text-only training.

Methodology

The researchers compared three types of models to evaluate their spatial reasoning capabilities:

  1. Text-only PLMs: BERT and RoBERTa.
  2. Vision-Language Pretrained Models (VL-PTMs): Models like VinVL, trained on aligned text-image pairs.
  3. Image Synthesis Models (ISMs): Generative models (e.g., VQGAN+CLIP) that create images from text prompts.

To probe these models, the authors used comparative prompts. For PLMs, they utilized masked language modeling (e.g., "A sofa is [MASK] than a mountain"). For ISMs, the models were prompted to generate a scene (e.g., "A sofa and a mountain"). The resulting images were then analyzed using automated object detectors (ISM-Box) and human evaluation (ISM-Human) to determine if the relative scales were correct.

Comparison of probing methods: PLMs and VL-PTMs use text masking, while ISMs generate images to reveal spatial relationships.

Key Results and Insights

Superior Accuracy and Consistency

The study found that ISMs significantly outperform text-based models in spatial accuracy. In size comparison tasks, human evaluation of ISM outputs reached 72.7% accuracy, compared to 54.1% for RoBERTa and 61.8% for the vision-language model VinVL. The performance gap was even wider in positional reasoning, where ISMs reached 73.4% accuracy while text-only models struggled at approximately 31%.

Furthermore, ISMs demonstrated much higher logical consistency. In symmetry tests (e.g., if a coin is smaller than a tire, then a tire is larger than a coin), ISMs maintained over 80% consistency, whereas PLMs often failed to correctly reverse the relationship, dropping to roughly 40%.

Visual Knowledge for NLU

The researchers proved that this implicit visual knowledge can be integrated back into natural language processing. By using an ISM to generate a "mental image" for a question-answering model (UnifiedQA), they observed measurable improvements in the model's ability to answer spatial commonsense questions across the board.

Conclusion

This research highlights that spatial commonsense is an emergent property of generative visual models. Unlike text models that rely on statistical co-occurrences of words, ISMs must learn a comprehensive internal representation of object scales and positions to reconstruct realistic scenes. While limitations exist—such as the difficulty of automatically parsing generated images—tapping into the visual signals of ISMs offers a promising path for building AI with a more human-like understanding of the physical world.

Summary updated

Abstract

Spatial commonsense, the knowledge about spatial position and relationship between objects (like the relative size of a lion and a girl, and the position of a boy relative to a bicycle when cycling), is an important part of commonsense knowledge. Although pretrained language models (PLMs) succeed in many NLP tasks, they are shown to be ineffective in spatial commonsense reasoning. Starting from the observation that images are more likely to exhibit spatial commonsense than texts, we explore whether models with visual signals learn more spatial commonsense than text-based PLMs. We propose a spatial commonsense benchmark that focuses on the relative scales of objects, and the positional relationship between people and objects under different actions. We probe PLMs and models with visual signals, including vision-language pretrained models and image synthesis models, on this benchmark, and find that image synthesis models are more capable of learning accurate and consistent spatial knowledge than other models. The spatial knowledge from image synthesis models also helps in natural language understanding tasks that require spatial commonsense.
BibTeX
@inproceedings{liu2022things,
  author = {Xiao Liu and Da Yin and Yansong Feng and Dongyan Zhao},
  title = {{Things not Written in Text: Exploring Spatial Commonsense from Visual Signals}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=eLMznlFpT2s},
}
Figure