Abstract
Spatial commonsense—knowing that a lion is larger than a person or that a cyclist sits on a bike—is fundamental to human understanding but rarely explicitly documented in text. This "reporting bias" leads text-only language models to struggle with basic physical reasoning. This paper investigates whether Image Synthesis Models (ISMs) can act as repositories for this missing knowledge. By generating images from text prompts, the authors show that ISMs implicitly capture consistent spatial signals that significantly outperform text-only models in spatial reasoning tasks.
The Challenge of Reporting Bias
Human communication is efficient; we rarely state the obvious. While a text might describe a "lion roaring" or define "cycling," it almost never mentions that a lion is much larger than a house cat or that a cyclist is positioned above the bike frame. This lack of explicit physical description creates a "reporting bias" in the corpora used to train Pretrained Language Models (PLMs) like BERT.
Because images must adhere to physical and geometric constraints to appear realistic, they offer a rich alternative source of spatial information. The authors hypothesize that models trained to generate images (ISMs) must internalize these constraints to produce coherent scenes.
Probing Spatial Knowledge
The researchers compared three model architectures to determine which best captures spatial relationships:
- Text-only PLMs (e.g., BERT, RoBERTa): Probed using masked language modeling (e.g., "A sofa is [MASK] than a mountain").
- Vision-Language Pretrained Models (VL-PTMs): Models like VinVL that align text and image features.
- Image Synthesis Models (ISMs): Specifically VQGAN+CLIP, which generates images from text prompts.
For ISMs, spatial knowledge was extracted by analyzing generated images. This was done via an automatic "ISM-Box" method (using object detectors and depth estimators) and human annotation ("ISM-Human").
Benchmarking Spatial Commonsense
To evaluate these models, the authors introduced a new benchmark focusing on:
- Size and Height: 500 object pairs across five scale levels (e.g., ant vs. mountain).
- Positional Relationships: 224 instances of humans interacting with objects (e.g., "driving a car" implies being inside).
- Robustness: A "stress test" using uncommon scenarios (e.g., "enchantress" instead of "woman") to ensure models aren't relying on simple text co-occurrences.
Key Findings
- Superior Accuracy: In size and height comparisons, human evaluation of ISM outputs reached 72.7% and 78.9% accuracy, respectively, significantly higher than RoBERTa's ~54%.
- Logical Consistency: ISMs demonstrated much higher symmetry (if A > B, then B < A) and transitivity. ISMs achieved 82.5% consistency, whereas text-only models ranged between 25% and 37%.
- Generative Advantage: Unlike discriminative models that label existing data, generative ISMs must actively construct relative scales. If a model draws a bird larger than a house, the CLIP component penalizes the mismatch with the conceptual scene, forcing the model to learn realistic proportions.
Limitations and Future Work
While promising, extracting knowledge from ISMs faces hurdles. Current object detectors often fail to parse generated images correctly; only 15% of ISM-generated images were successfully processed by the automatic bounding-box method. Furthermore, image quality can be inconsistent. However, as synthesis models like Stable Diffusion improve, they are expected to become even more effective "spatial world simulators," providing a vital bridge between language and the physical world.
Summary updated