Abstract: Learning representations of spatial references in natural language is a key challenge in tasks like autonomous navigation and robotic manipulation. Recent work has investigated various neural architectures for learning multi-modal representations for spatial concepts. However, the lack of explicit reasoning over entities makes such approaches vulnerable to noise in input text or state observations. In this paper, we develop effective models for understanding spatial references in text that are robust and interpretable, without sacrificing performance. We design a text-conditioned relation network whose parameters are dynamically computed with a cross-modal attention module to capture fine-grained spatial relations between entities. This design choice provides interpretability of learned intermediate outputs. Experiments across three tasks demonstrate that our model achieves superior performance, with a 17% improvement in predicting goal locations and a 15% improvement in robustness compared to state-of-the-art systems.
Introduction
Grounding spatial references is a deceptively complex task for autonomous systems. For a robot to follow an instruction such as "reach the cell below the westernmost circle," it must perform several discrete reasoning steps: identify all circles in the environment, determine which one is furthest west, and then calculate the specific coordinates relative to that object.
Most existing deep learning models treat this as a "black box" fusion problem, concatenating text and image features into a high-dimensional vector. While these models can perform well on specific datasets, they suffer from two major flaws: fragility to noise (such as irrelevant objects or filler words) and a lack of interpretability, making them difficult to trust in safety-critical robotic applications.
Proposed Architecture: Text-Conditioned Relation Network
To address these limitations, we propose a Text-Conditioned Relation Network (RNet). The architecture explicitly models pairwise relationships between entities in an environment through three main components:
- Text Encoder with Cross-Modal Attention: Instead of a static text representation, the model uses an LSTM that attends to specific words based on the visual entities being considered.
- Dynamic Parameter Generation: Rather than utilizing fixed weights, the parameters of the Relation Network are computed on-the-fly based on the input text. The instruction effectively acts as a "program" that configures the neural network to look for specific spatial patterns.
- Pairwise Relation Scoring: The model iterates through pairs of cells in the environment map, calculating a "relation score" for each pair. This mimics human reasoning: "Is object A the reference point? If so, is object B in the correct relative position?"
By focusing on pairs of entities, the model naturally factorizes the problem into identifying reference objects and inferring directions.
Experimental Setup
The model was evaluated across three distinct tasks using established benchmarks:
- ShapeWorld: A dataset of abstract shapes used for a classification task (determining if a spatial statement about the shapes is True or False).
- PuddleWorld: A 2D grid navigation environment where an agent must find goals based on local ("two cells left of the triangle") or global ("the westernmost rock") instructions.
- ISI (Block World): A complex environment containing blocks marked with corporate logos, requiring precise relative positioning.
Key Results and Discussion
The proposed model, t-RNetAttn, outperformed state-of-the-art baselines (such as t-CNN and LingUNet) across all metrics:
- Goal Localization: The model achieved a 17% improvement in accurately predicting the coordinates of a goal described in text.
- Navigation: In Reinforcement Learning tasks, the model showed a 13.5% relative improvement in policy quality.
- Robustness: When subjected to "textual noise" (random irrelevant words) or "observational noise" (unseen objects), baseline models' performance plummeted by 20-45%. In contrast, the RNet model's performance only degraded by approximately 10%.
- Interpretability: By visualizing internal "relation maps," we confirmed the model was correctly identifying reference objects (e.g., a specific rock or spade) rather than relying on global dataset statistics.
Conclusion
Our research demonstrates that using text to dynamically generate the weights of a vision network is superior to simple feature concatenation. This approach allows the model to "re-wire" itself for every unique instruction, providing both robustness through sparsity and inherent interpretability. Future work will focus on integrating explicit object detectors to allow the RNet to operate directly on complex, pixel-level real-world visual scenes and expanding the range of spatial relations beyond immediate neighborhoods.
Summary updated