Things not Written in Text: Exploring Spatial Commonsense from Visual Signals

Xiao Liu, Da Yin, Yansong Feng, Dongyan Zhao

Abstract

Spatial commonsense—knowing that a lion is larger than a person or that a cyclist sits on a bike—is fundamental to human understanding but rarely explicitly documented in text. This "reporting bias" leads text-only language models to struggle with basic physical reasoning. This paper investigates whether Image Synthesis Models (ISMs) can act as repositories for this missing knowledge. By generating images from text prompts, the authors show that ISMs implicitly capture consistent spatial signals that significantly outperform text-only models in spatial reasoning tasks.

The Challenge of Reporting Bias

Human communication is efficient; we rarely state the obvious. While a text might describe a "lion roaring" or define "cycling," it almost never mentions that a lion is much larger than a house cat or that a cyclist is positioned above the bike frame. This lack of explicit physical description creates a "reporting bias" in the corpora used to train Pretrained Language Models (PLMs) like BERT.

Comparison of spatial information in text vs. images for size and positional relationships.

Because images must adhere to physical and geometric constraints to appear realistic, they offer a rich alternative source of spatial information. The authors hypothesize that models trained to generate images (ISMs) must internalize these constraints to produce coherent scenes.

Probing Spatial Knowledge

The researchers compared three model architectures to determine which best captures spatial relationships:

  1. Text-only PLMs (e.g., BERT, RoBERTa): Probed using masked language modeling (e.g., "A sofa is [MASK] than a mountain").
  2. Vision-Language Pretrained Models (VL-PTMs): Models like VinVL that align text and image features.
  3. Image Synthesis Models (ISMs): Specifically VQGAN+CLIP, which generates images from text prompts.
Probing spatial commonsense across different model types: PLM, VL-PTM, and ISM.

For ISMs, spatial knowledge was extracted by analyzing generated images. This was done via an automatic "ISM-Box" method (using object detectors and depth estimators) and human annotation ("ISM-Human").

Benchmarking Spatial Commonsense

To evaluate these models, the authors introduced a new benchmark focusing on:

  • Size and Height: 500 object pairs across five scale levels (e.g., ant vs. mountain).
  • Positional Relationships: 224 instances of humans interacting with objects (e.g., "driving a car" implies being inside).
  • Robustness: A "stress test" using uncommon scenarios (e.g., "enchantress" instead of "woman") to ensure models aren't relying on simple text co-occurrences.

Key Findings

  • Superior Accuracy: In size and height comparisons, human evaluation of ISM outputs reached 72.7% and 78.9% accuracy, respectively, significantly higher than RoBERTa's ~54%.
  • Logical Consistency: ISMs demonstrated much higher symmetry (if A > B, then B < A) and transitivity. ISMs achieved 82.5% consistency, whereas text-only models ranged between 25% and 37%.
  • Generative Advantage: Unlike discriminative models that label existing data, generative ISMs must actively construct relative scales. If a model draws a bird larger than a house, the CLIP component penalizes the mismatch with the conceptual scene, forcing the model to learn realistic proportions.

Limitations and Future Work

While promising, extracting knowledge from ISMs faces hurdles. Current object detectors often fail to parse generated images correctly; only 15% of ISM-generated images were successfully processed by the automatic bounding-box method. Furthermore, image quality can be inconsistent. However, as synthesis models like Stable Diffusion improve, they are expected to become even more effective "spatial world simulators," providing a vital bridge between language and the physical world.

Summary updated

Abstract

Spatial commonsense, the knowledge about spatial position and relationship between objects (like the relative size of a lion and a girl, and the position of a boy relative to a bicycle when cycling), is an important part of commonsense knowledge. Although pretrained language models (PLMs) succeed in many NLP tasks, they are shown to be ineffective in spatial commonsense reasoning. Starting from the observation that images are more likely to exhibit spatial commonsense than texts, we explore whether models with visual signals learn more spatial commonsense than text-based PLMs. We propose a spatial commonsense benchmark that focuses on the relative scales of objects, and the positional relationship between people and objects under different actions. We probe PLMs and models with visual signals, including vision-language pretrained models and image synthesis models, on this benchmark, and find that image synthesis models are more capable of learning accurate and consistent spatial knowledge than other models. The spatial knowledge from image synthesis models also helps in natural language understanding tasks that require spatial commonsense.
BibTeX
@inproceedings{liu2022things,
  author = {Xiao Liu and Da Yin and Yansong Feng and Dongyan Zhao},
  title = {{Things not Written in Text: Exploring Spatial Commonsense from Visual Signals}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=eLMznlFpT2s},
}
Figure