Abstract
Spatial Description Resolution (SDR) is the task of localizing a target in a 360-degree street-view panorama based on natural language instructions. We introduce SIRI (Spatial Relation Induced Network), a framework designed to overcome the challenges of complex urban entities and long-range spatial reasoning. By integrating object-level visual feature correlation, local spatial relation guided distillation, and global spatial positional embedding, SIRI achieves a 58.33% accuracy on the Touchdown benchmark, outperforming previous state-of-the-art models by approximately 24% in relative terms.
Introduction
Locating a specific point in a dense urban environment using natural language is a complex task for artificial intelligence. Unlike standard visual grounding in perspective images, Spatial Description Resolution (SDR) in 360-degree panoramas involves resolving long-range dependencies and navigating through multiple spatial "hops." For example, a user might specify a target relative to a sequence of landmarks: "There is a building with a red outer wall on your left... touchdown is at the center of the sign."
Methodology
The SIRI architecture addresses three core challenges of SDR: entity complexity, spatial reasoning, and panoramic ambiguity.
Object-level Visual Feature Correlation
To avoid the pitfalls of pre-trained object detectors in cluttered urban scenes, SIRI employs Global Reasoning (GloRe) units. These units project visual features into a coordinate-independent latent space to model relationships between implicit objects. This allows the network to understand contextual hierarchies, such as the relationship between a restaurant sign and the building facade, without explicit bounding boxes.
Local Spatial Relation Guided Distillation
SIRI utilizes a "linguistic switch" mechanism to process spatial instructions. The model identifies high-frequency orientation words (e.g., left, right, front) in the text. These words act as triggers to activate specific convolutional branches that distill visual features, focusing the model's attention on regions that correspond to the described spatial direction.
Global Spatial Positional Embedding
To resolve global positional ambiguities inherent in 360-degree views, SIRI incorporates global coordinate maps. These maps provide an egocentric reference frame centered at the agent's position (the bottom-center of the panorama). This "sense of self" is crucial for distinguishing between multiple identical objects (e.g., two different mailboxes) based on their relative position to the viewer.
Results and Discussion
The researchers evaluated SIRI on the Touchdown dataset and a new Extended Touchdown dataset designed to test generalization.
On the primary Touchdown benchmark, SIRI achieved an accuracy of 58.33% (within an 80-pixel radius), a significant improvement over the previous state-of-the-art LingUnet model, which reached 34.59%. This represents a relative performance boost of nearly 24%. Furthermore, SIRI demonstrated superior robustness in "copy-paste" tests involving visual twins, confirming that the model effectively utilizes spatial instructions rather than relying solely on object appearance.
While SIRI offers high accuracy, it currently faces a trade-off in computational efficiency, taking approximately 2.5 times longer to process images than simpler models. Future work will focus on model compression to enable real-time applications in mobile robotics and augmented reality.
Reference: Wang, P., Luo, W., Xu, Y., et al. (2020). SIRI: Spatial Relation Induced Network For Spatial Description Resolution. arXiv:2010.14301v1 [cs.CV].
Summary updated