Abstract
Learning representations of spatial references in natural language is a key challenge in tasks like autonomous navigation and robotic manipulation. Recent work has investigated various neural architectures for learning multi-modal representations for spatial concepts. However, the lack of explicit reasoning over entities makes such approaches vulnerable to noise in input text or state observations. In this paper, we develop effective models for understanding spatial references in text that are robust and interpretable, without sacrificing performance. We design a text-conditioned relation network whose parameters are dynamically computed with a cross-modal attention module to capture fine-grained spatial relations between entities. Experiments across three tasks demonstrate that our model achieves superior performance, with a 17% improvement in predicting goal locations and a 15% improvement in robustness compared to state-of-the-art systems.
Introduction
Grounding spatial references—the ability to link natural language to physical locations—is a cornerstone of robotics and autonomous navigation. However, current deep learning models often treat this as a "black box" pattern-matching problem. Standard Convolutional Neural Networks (CNNs) tend to capture global patterns but struggle with fine-grained reasoning between specific entities. This makes them "brittle": performance often collapses when irrelevant objects are added to a scene or extra words are inserted into instructions. Furthermore, the lack of transparency in standard models makes them difficult to trust in real-world applications.
Methodology: Text-Conditioned Relation Networks
The proposed architecture prioritizes explicit reasoning over entities and their relationships through three main components:
- Text Encoder (LSTM + Attention): This module processes instructions using a cross-modal attention mechanism to focus on specific words based on the visual entities present in the environment.
- Text-Conditioned Relation Network (RNet): Instead of using fixed weights, the RNet's parameters are dynamically computed from the text representation. This ensures the model's reasoning logic (e.g., "above" vs. "west of") is inherently tied to the instruction. By evaluating relationships between pairs of entities rather than looking at the whole image at once, the model can ignore "distractor" objects not mentioned in the text.
- Task-Dependent Output Network: Depending on the task, this produces a classification (True/False), a value map for navigation, or an action policy for Reinforcement Learning.
Experimental Setup and Datasets
The model was evaluated across three distinct benchmarks to test various types of spatial reasoning:
- ShapeWorld: A dataset of abstract shapes used for classification tasks (e.g., "Is the statement 'A magenta triangle is to the left of a magenta pentagon' true?").
- PuddleWorld: A 10x10 grid world where an agent must navigate to a goal based on local or global instructions.
- ISI (Instruction Following): A complex environment involving moving blocks with corporate logos to specific relative positions.
Key Results and Insights
The proposed model, t-RNetAttn, consistently outperformed state-of-the-art baselines like t-CNN and t-UVFA:
- Localization & Navigation: It achieved a 17% improvement in predicting goal coordinates and up to a 13.5% relative improvement in Reinforcement Learning policy quality.
- Robustness to Noise: When 10 "noise" objects were added to the map, the RNet's error increased by only ~30%, compared to over 45% for standard CNN baselines. It showed similar resilience to textual noise (random words in instructions).
- Interpretability: By calculating explicit relation scores, the model generates "Relation Maps" that visualize which objects were identified as referents (landmarks) and goals, providing a clear window into the AI's reasoning process.
Conclusion and Future Work
By using text to generate the weights of a Relation Network, this research demonstrates that spatial reasoning can be both high-performing and interpretable. While currently focused on local neighborhoods and discrete entities, future work aims to scale these networks to larger 3D observation spaces and complex first-person vision tasks.
Summary updated