Robust and Interpretable Grounding of Spatial References with Relation Networks

Tsung-Yen Yang, Karthik Narasimhan

Abstract

Learning representations of spatial references in natural language is a key challenge in tasks like autonomous navigation and robotic manipulation. Recent work has investigated various neural architectures for learning multi-modal representations for spatial concepts. However, the lack of explicit reasoning over entities makes such approaches vulnerable to noise in input text or state observations. In this paper, we develop effective models for understanding spatial references in text that are robust and interpretable, without sacrificing performance. We design a text-conditioned relation network whose parameters are dynamically computed with a cross-modal attention module to capture fine-grained spatial relations between entities. Experiments across three tasks demonstrate that our model achieves superior performance, with a 17% improvement in predicting goal locations and a 15% improvement in robustness compared to state-of-the-art systems.

Introduction

Grounding spatial references—the ability to link natural language to physical locations—is a cornerstone of robotics and autonomous navigation. However, current deep learning models often treat this as a "black box" pattern-matching problem. Standard Convolutional Neural Networks (CNNs) tend to capture global patterns but struggle with fine-grained reasoning between specific entities. This makes them "brittle": performance often collapses when irrelevant objects are added to a scene or extra words are inserted into instructions. Furthermore, the lack of transparency in standard models makes them difficult to trust in real-world applications.

Methodology: Text-Conditioned Relation Networks

The proposed architecture prioritizes explicit reasoning over entities and their relationships through three main components:

  1. Text Encoder (LSTM + Attention): This module processes instructions using a cross-modal attention mechanism to focus on specific words based on the visual entities present in the environment.
  2. Text-Conditioned Relation Network (RNet): Instead of using fixed weights, the RNet's parameters are dynamically computed from the text representation. This ensures the model's reasoning logic (e.g., "above" vs. "west of") is inherently tied to the instruction. By evaluating relationships between pairs of entities rather than looking at the whole image at once, the model can ignore "distractor" objects not mentioned in the text.
  3. Task-Dependent Output Network: Depending on the task, this produces a classification (True/False), a value map for navigation, or an action policy for Reinforcement Learning.

Experimental Setup and Datasets

The model was evaluated across three distinct benchmarks to test various types of spatial reasoning:

  • ShapeWorld: A dataset of abstract shapes used for classification tasks (e.g., "Is the statement 'A magenta triangle is to the left of a magenta pentagon' true?").
  • PuddleWorld: A 10x10 grid world where an agent must navigate to a goal based on local or global instructions.
  • ISI (Instruction Following): A complex environment involving moving blocks with corporate logos to specific relative positions.
Example of a navigation task in PuddleWorld where the agent must reach a cell relative to the westernmost circle.
ShapeWorld classification tasks require the model to determine if a spatial statement accurately describes the scene.

Key Results and Insights

The proposed model, t-RNetAttn, consistently outperformed state-of-the-art baselines like t-CNN and t-UVFA:

  • Localization & Navigation: It achieved a 17% improvement in predicting goal coordinates and up to a 13.5% relative improvement in Reinforcement Learning policy quality.
  • Robustness to Noise: When 10 "noise" objects were added to the map, the RNet's error increased by only ~30%, compared to over 45% for standard CNN baselines. It showed similar resilience to textual noise (random words in instructions).
  • Interpretability: By calculating explicit relation scores, the model generates "Relation Maps" that visualize which objects were identified as referents (landmarks) and goals, providing a clear window into the AI's reasoning process.

Conclusion and Future Work

By using text to generate the weights of a Relation Network, this research demonstrates that spatial reasoning can be both high-performing and interpretable. While currently focused on local neighborhoods and discrete entities, future work aims to scale these networks to larger 3D observation spaces and complex first-person vision tasks.

Summary updated

Abstract

Learning representations of spatial references in natural language is a key challenge in tasks like autonomous navigation and robotic manipulation. Recent work has investigated various neural architectures for learning multi-modal representations for spatial concepts. However, the lack of explicit reasoning over entities makes such approaches vulnerable to noise in input text or state observations. In this paper, we develop effective models for understanding spatial references in text that are robust and interpretable, without sacrificing performance. We design a text-conditioned \textit{relation network} whose parameters are dynamically computed with a cross-modal attention module to capture fine-grained spatial relations between entities. This design choice provides interpretability of learned intermediate outputs. Experiments across three tasks demonstrate that our model achieves superior performance, with a 17\% improvement in predicting goal locations and a 15\% improvement in robustness compared to state-of-the-art systems.
BibTeX
@inproceedings{yang2020robust,
  author = {Tsung-Yen Yang and Karthik Narasimhan},
  title = {{Robust and Interpretable Grounding of Spatial References with Relation Networks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=8StDlaEKLSR},
}
Figure