Abstract
Traditional 3D scene understanding often focuses on isolated object identification, neglecting the functional context provided by spatial arrangements. We present TB-HSU (Transformer-Based Hierarchical Scene Understanding), a model designed to infer contextual affordances by mapping objects to functional regions and room types. Utilizing a novel 3D Hierarchical Scene Graph (3DHSG) dataset, we demonstrate that TB-HSU significantly improves room and region classification accuracy. Furthermore, we show that providing this hierarchical context to Large Language Models (LLMs) like GPT-4o enhances their ability to perform common-sense reasoning in complex indoor environments.
Introduction: Beyond Object Labels
In computer vision, object detection has reached high levels of maturity—identifying a "chair" or a "table" is now standard. However, for autonomous agents to interact meaningfully with their environment, they must understand affordances: the potential actions an object supports.
Crucially, affordances are context-dependent. A chair's purpose changes based on its surroundings—it facilitates "dining" when placed at a table, but "resting" when positioned in front of a television. Existing datasets often treat these as static labels. TB-HSU addresses this by modeling how spatial context (the "region" or "room") defines the functional purpose of objects.
Methodology: The TB-HSU Framework
The core of our approach is the 3D Hierarchical Scene Graph (3DHSG), which structures indoor environments into three layers:
- Objects: Individual items with semantic labels and 3D coordinates.
- Regions: Functional zones (e.g., "workstation," "dining area") that cluster related objects.
- Rooms: The overarching category (e.g., "kitchen," "bedroom").
The TB-HSU model employs a Transformer encoder to process a sequence of object tokens. Each token integrates a semantic embedding (from the object label) and a position embedding (calculated from the object's centroid relative to the room's center). This multi-task learning framework simultaneously predicts room types and region-specific affordances.
Datasets and Benchmarking
We introduce the 3DHSG Dataset, built upon 3RScan and 3DSSG, featuring 120 scene graphs with nearly 4,000 nodes. It provides ground truth for 12 room types, 27 region affordances, and 87 object affordances. Performance was also validated on standard benchmarks including ScanNet (20 & 200) and Matterport3D.
Key Insights and LLM Integration
Our research highlights that while room classification relies heavily on object presence, region classification is highly sensitive to spatial arrangement (position embeddings).
A significant finding is the synergy between TB-HSU and Large Language Models. When tasked with locating items based on common-sense reasoning (e.g., "Where would I find shampoo?"), standard LLMs like GPT-4o struggle with raw point cloud data. However, when provided with the hierarchical graph generated by TB-HSU, the LLM's reasoning becomes significantly more logical and human-like, as it can leverage the identified functional zones.
Results
TB-HSU consistently outperforms baseline models:
- Room Classification: Achieved 91.67% accuracy on the 3DHSG dataset, a ~4.5% improvement over the Hydra model on Matterport3D.
- Region Classification: Reached 87.27% accuracy, vastly outperforming MLP baselines (44.56%), demonstrating the Transformer's superior ability to capture inter-object relationships.
Conclusion and Future Work
TB-HSU demonstrates that contextual hierarchy is vital for functional scene understanding. Future work will focus on expanding object variety, refining the distinction between similar functional regions (e.g., "study" vs. "work"), and integrating this framework into real-time robotic navigation systems.
Summary updated