SIRI: Spatial Relation Induced Network For Spatial Description Resolution

Peiyao Wang, Weixin Luo, Yanyu Xu, Haojie Li, Shugong Xu, Jianyu Yang, Shenghua Gao

Abstract

Spatial Description Resolution (SDR) is the task of localizing a target in a 360-degree street-view panorama based on natural language instructions. We introduce SIRI (Spatial Relation Induced Network), a framework designed to overcome the challenges of complex urban entities and long-range spatial reasoning. By integrating object-level visual feature correlation, local spatial relation guided distillation, and global spatial positional embedding, SIRI achieves a 58.33% accuracy on the Touchdown benchmark, outperforming previous state-of-the-art models by approximately 24% in relative terms.

Introduction

Locating a specific point in a dense urban environment using natural language is a complex task for artificial intelligence. Unlike standard visual grounding in perspective images, Spatial Description Resolution (SDR) in 360-degree panoramas involves resolving long-range dependencies and navigating through multiple spatial "hops." For example, a user might specify a target relative to a sequence of landmarks: "There is a building with a red outer wall on your left... touchdown is at the center of the sign."

An example of the SDR task where a target must be located within a panoramic street view based on linguistic cues.

Methodology

The SIRI architecture addresses three core challenges of SDR: entity complexity, spatial reasoning, and panoramic ambiguity.

Object-level Visual Feature Correlation

To avoid the pitfalls of pre-trained object detectors in cluttered urban scenes, SIRI employs Global Reasoning (GloRe) units. These units project visual features into a coordinate-independent latent space to model relationships between implicit objects. This allows the network to understand contextual hierarchies, such as the relationship between a restaurant sign and the building facade, without explicit bounding boxes.

Local Spatial Relation Guided Distillation

SIRI utilizes a "linguistic switch" mechanism to process spatial instructions. The model identifies high-frequency orientation words (e.g., left, right, front) in the text. These words act as triggers to activate specific convolutional branches that distill visual features, focusing the model's attention on regions that correspond to the described spatial direction.

Global Spatial Positional Embedding

To resolve global positional ambiguities inherent in 360-degree views, SIRI incorporates global coordinate maps. These maps provide an egocentric reference frame centered at the agent's position (the bottom-center of the panorama). This "sense of self" is crucial for distinguishing between multiple identical objects (e.g., two different mailboxes) based on their relative position to the viewer.

The SIRI network architecture, illustrating the flow from image and text input to the final heatmap prediction via GloRe units and spatial distillation.

Results and Discussion

The researchers evaluated SIRI on the Touchdown dataset and a new Extended Touchdown dataset designed to test generalization.

On the primary Touchdown benchmark, SIRI achieved an accuracy of 58.33% (within an 80-pixel radius), a significant improvement over the previous state-of-the-art LingUnet model, which reached 34.59%. This represents a relative performance boost of nearly 24%. Furthermore, SIRI demonstrated superior robustness in "copy-paste" tests involving visual twins, confirming that the model effectively utilizes spatial instructions rather than relying solely on object appearance.

While SIRI offers high accuracy, it currently faces a trade-off in computational efficiency, taking approximately 2.5 times longer to process images than simpler models. Future work will focus on model compression to enable real-time applications in mobile robotics and augmented reality.

Reference: Wang, P., Luo, W., Xu, Y., et al. (2020). SIRI: Spatial Relation Induced Network For Spatial Description Resolution. arXiv:2010.14301v1 [cs.CV].

Summary updated

Abstract

Spatial Description Resolution, as a language-guided localization task, is proposed for target location in a panoramic street view, given corresponding language descriptions. Explicitly characterizing an object-level relationship while distilling spatial relationships are currently absent but crucial to this task. Mimicking humans, who sequentially traverse spatial relationship words and objects with a first-person view to locate their target, we propose a novel spatial relationship induced (SIRI) network. Specifically, visual features are firstly correlated at an implicit object-level in a projected latent space; then they are distilled by each spatial relationship word, resulting in each differently activated feature representing each spatial relationship. Further, we introduce global position priors to fix the absence of positional information, which may result in global positional reasoning ambiguities. Both the linguistic and visual features are concatenated to finalize the target localization. Experimental results on the Touchdown show that our method is around 24\% better than the state-of-the-art method in terms of accuracy, measured by an 80-pixel radius. Our method also generalizes well on our proposed extended dataset collected using the same settings as Touchdown.
BibTeX
@inproceedings{wang2020siri,
  author = {Peiyao Wang and Weixin Luo and Yanyu Xu and Haojie Li and Shugong Xu and Jianyu Yang and Shenghua Gao},
  title = {{SIRI: Spatial Relation Induced Network For Spatial Description Resolution}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure