Introduction
In autonomous robotics, identifying objects within a scene is a fundamental task, yet it is insufficient for complex manipulation. For a robot to successfully execute a command like "pick up the book," it must understand the physical hierarchy and spatial constraints of its environment. If the target book is obstructed by another object, a standard object detector fails to provide the necessary context for a safe and logical sequence of actions. This research addresses the "spatial gap" by introducing a specialized dataset and benchmarking suite designed to enhance robotic scene understanding through spatial relationship awareness.
The Spatial Relationship Aware Dataset
The primary contribution of this work is the Spatial Relationship Aware Dataset for Robotics. Unlike general-purpose datasets such as Visual Genome, this collection is tailored specifically for indoor robotic applications.
- Data Acquisition: The dataset consists of nearly 1,000 images captured using a Boston Dynamics Spot robot. This provides realistic, low-angle perspectives that accurately reflect the viewpoint of a mobile robot navigating an indoor environment.
- Annotation Density: Each image contains 5–10 common indoor objects (e.g., books, bottles, cubes, remotes). Annotations include precise bounding boxes, object attributes (color, material, shape), and an average of ten spatial relationships per image.
- Spatial Predicates: The dataset focuses on seven key spatial relations: behind, in front of, on, to the left of, to the right of, under, and near.
To facilitate the creation of this high-quality data, the researchers utilized a custom annotation tool designed for efficient labeling of complex scene graphs and object attributes.
Methodology
The researchers proposed a two-stage pipeline to bridge the gap between raw visual input and actionable task plans.
- Object Detection: A YOLOv10m backbone was employed for its efficiency and high accuracy in identifying objects within the robot's field of view.
- Scene Graph Generation (SGG): The study benchmarked six state-of-the-art relation-prediction heads: PE-NET, VCTree Predictor, REACT Predictor, Motif Predictor, Causal Analysis Predictor, and Transformer Predictor. These models transform detected objects into a structured graph of spatial relationships.
Enhancing Task Planning with Spatial Context
The ultimate goal of spatial awareness is to improve robotic planning. Large Language Models (LLMs) like ChatGPT-4o, while powerful, often generate unexecutable plans when relying solely on raw image data or simple object lists. They may attempt to move an object that is physically pinned under another because they lack a structured understanding of the scene's hierarchy.
By providing the LLM with an explicit scene graph—a structured list of "Predicted Spatial Relationships"—the model gains a "spatial anchor." This allows the LLM to reason about physical constraints, leading to the generation of valid, multi-step execution plans that account for obstructions.
Results and Discussion
The benchmarking results revealed a clear trade-off between accuracy and real-time performance:
- Detection: The YOLOv10m model achieved a robust mAP@50 of 0.93.
- Relational Accuracy: The VCTree Predictor led in accuracy with a Recall@100 of 0.4840, though its high latency (92.5 ms) may limit its use in high-speed applications.
- Real-Time Performance: The Transformer and Motif Predictors offered a better balance for robotics, achieving "real-time" speeds (under 30 ms) with competitive recall scores (~0.45–0.48).
Qualitative analysis confirmed that structured spatial data is the "missing link" for LLM-based planning, enabling models to correctly resolve object hierarchies that are otherwise ambiguous in 2D images.
Conclusion
This work demonstrates that explicit spatial relationship awareness is critical for moving beyond simple object recognition toward intelligent robotic manipulation. By providing a specialized dataset and benchmarking SGG models, the authors offer a foundation for developing robots that can truly understand the physical constraints of their world. Future work will focus on expanding scene diversity and refining the granularity of spatial predicates.
Summary updated