Abstract
Generating 2D spatial layouts from natural language descriptions requires a deep understanding of both explicit instructions and implicit spatial common sense. We propose SR-BERT (Spatial-Reasoning BERT), a model that utilizes Masked Position Modelling (MPM) to arrange objects in a scene. Unlike previous methods that rely on rigid priors or simple pairwise relations, SR-BERT employs non-autoregressive, iterative decoding to generate complex scenes from nuanced natural language. Evaluated on the Abstract Scenes dataset, SR-BERT achieves a position similarity of 0.512, significantly outperforming existing benchmarks. Human evaluations confirm that 74.6% of generated relations are spatially accurate, demonstrating the model's capacity for robust spatial reasoning and compositional generalization.
Introduction
While humans possess an innate "spatial common sense"—understanding, for instance, that characters looking at an object should be oriented toward it—machines often struggle to interpret these nuances from natural language. Previous attempts at automated scene generation have been hampered by three primary issues: a reliance on strong priors in highly predictable environments (like indoor rooms), a focus on simple pairwise relationships between objects, and a dependency on rigid symbolic inputs rather than fluid natural language.
This research introduces SR-BERT to address these gaps. The goal is to take a set of objects and a descriptive paragraph and automatically determine the optimal 2D coordinates and orientations for every element in the scene.
Methodology
The core innovation is the adaptation of the pre-trained BERT-base architecture to a multi-modal task involving sequential text and spatial object data.
Spatial Embeddings and Masked Position Modelling (MPM)
Each object in a scene is represented by a unique index, its $[x, y]$ coordinates, and a binary orientation indicator. Inspired by BERT’s Masked Language Modelling, we introduce Masked Position Modelling (MPM). During training, spatial attributes of objects are masked, forcing the model to predict missing positions and orientations based on the accompanying text and the locations of unmasked objects.
Non-Autoregressive Decoding
Unlike traditional autoregressive models that generate output sequentially, SR-BERT can predict multiple object positions simultaneously. The researchers explored various decoding strategies, finding that iterative refinement—specifically Highest Confidence (HC) and Human-order (HO) (placing background elements before characters)—produced the most coherent layouts. The model was implemented in both continuous (regression-based) and discrete (quantized grid-based) versions.
Experimental Setup
The model was evaluated using the Abstract Scenes dataset, which contains over 10,000 scenes featuring 126 unique clip-art types. Each scene is paired with approximately six descriptive sentences. The dataset was split to ensure the model was tested on semantically novel descriptions not seen during training.
Results
SR-BERT demonstrated superior performance across multiple benchmarks. In a full generation pipeline, it achieved an absolute position similarity of 0.512, a marked improvement over previous state-of-the-art models (0.409–0.449).
Qualitative analysis and human validation further support these findings. In a user study, 74.6% of the spatial relations in scenes generated by the discrete SR-BERT model were rated as "spatially true" relative to the text, nearing the 88% accuracy of human-authored ground truth. The model also exhibited strong compositional generalization, successfully arranging objects in novel combinations it had not encountered during training.
Discussion and Conclusion
The success of SR-BERT highlights the importance of bidirectionality and implicit reasoning in spatial tasks. By allowing every object to "attend" to every other object and every word in the description, the model captures relative spatial cues (e.g., placing a character further away if the text mentions they are "afraid").
Future work will look to extend this framework to 3D environments, which present additional challenges such as depth and occlusion, and to move beyond a fixed vocabulary of clip-art categories toward more open-ended object generation.
Summary updated