TB-HSU: Hierarchical 3D Scene Understanding with Contextual Affordances

Wenting Xu, Viorela Ila, Luping Zhou, Craig T. Jin

Introduction

In the fields of robotics and computer vision, the concept of "affordance"—the potential actions an agent can perform with an object—is fundamental to interaction. Traditionally, 3D scene understanding has treated affordances as static, intrinsic properties of objects (e.g., a chair is for sitting). However, this approach fails to capture the nuance of real-world environments where an object's function is often defined by its context. A chair positioned at a dining table serves a different purpose than one placed in a relaxation nook.

To address this, researchers have introduced TB-HSU (Transformer-Based Hierarchical Scene Understanding), a model designed to construct hierarchical 3D scene graphs that account for context-dependent affordances. By mapping environments across three distinct scales—Objects, Regions, and Rooms—TB-HSU enables more sophisticated spatial reasoning for autonomous agents.

Hierarchical Representation and Contextual Affordance

The core of the proposed approach is the 3D Hierarchical Scene Graph (3DHSG). This structure organizes spatial data into three layers, allowing for what the authors term "contextual zooming."

Hierarchical 3D Scene Graph (3DHSG) structure mapping objects to functional regions and room types.
  1. Layer 1 (Objects): The most granular level, representing individual items with semantic labels (e.g., "table," "lamp") and spatial centroids.
  2. Layer 2 (Regions): Intermediate clusters of objects that share a functional purpose. For instance, a group of chairs and a table might constitute a "dining region," while a sofa and television form a "resting region."
  3. Layer 3 (Rooms): The highest level of abstraction, classifying the overall space (e.g., "Living Room," "Kitchen").

This hierarchy allows the model to refine affordance labels based on the surrounding environment. While a bed's general affordance is "sleeping," zooming into the "bed region" allows for more specific affordances like "propping" for pillows or "lying on" for the mattress.

The TB-HSU Architecture

The TB-HSU model utilizes a Transformer-based architecture to process these hierarchical relationships. Unlike previous methods that might rely solely on visual features, TB-HSU leverages both semantic and spatial information.

The TB-HSU architecture utilizing semantic and position embeddings within a Transformer-based framework.

Input Embeddings

The model processes two primary inputs for each object in a scene:

  • Semantic Embeddings: These are derived from object labels using either one-hot vectors or CLIP-based embeddings, capturing the inherent identity of the object.
  • Position Embeddings: These represent the spatial relationship between an object's centroid and the center of the room. This spatial context is vital for identifying functional regions.

Multi-Task Learning

The combined embeddings are fed into a Transformer encoder. The model is then trained via multi-task learning to simultaneously perform room classification and identify functional regions. A key insight from the research is that while room classification depends heavily on the "bag of objects" (semantics), region identification is significantly more dependent on position embeddings, as the relative placement of objects defines their collective function.

Experimental Results and LLM Integration

The researchers evaluated TB-HSU against several benchmarks, including ScanNet and Matterport3D, and introduced a new specialized dataset, the 3DHSG Dataset, which contains 120 scene graphs with nearly 4,000 nodes annotated for context-aware affordances.

Performance Benchmarks

On the Matterport3D dataset, TB-HSU achieved a room classification accuracy of 62.19%, notably outperforming the previous state-of-the-art model, Hydra (57.67%). On the custom 3DHSG dataset, the model demonstrated high precision, reaching 91.67% accuracy for room classification and 87.27% accuracy for region identification. The model also showed strong scalability, with performance improving as the granularity of object labels increased (e.g., moving from ScanNet20 to ScanNet200).

Augmenting Large Language Models

One of the most compelling applications of TB-HSU is its ability to enhance the spatial reasoning of Large Language Models (LLMs) like GPT-4o. By providing the LLM with a structured 3DHSG rather than raw data, the researchers found that the model could more accurately predict the location of "hidden" objects. For example, when asked to find shampoo, an LLM equipped with the hierarchical graph could logically narrow its search to the "shower region" or "washing region" within a bathroom, demonstrating a level of common-sense spatial reasoning previously difficult to achieve.

Conclusion and Future Directions

TB-HSU represents a significant step forward in bridging the gap between low-level object detection and high-level scene understanding. By formalizing the relationship between context and affordance, it provides a robust framework for robotic interaction in complex indoor environments.

Future work will focus on expanding the variety of objects within the training datasets to improve generalization and addressing the ambiguity between similar functional regions (e.g., distinguishing a "study nook" from a general "workspace"). Additionally, the authors aim to explore dynamic environments, investigating how these hierarchical graphs can adapt as objects are moved in real-time.

Summary updated

BibTeX
@inproceedings{xu2025tb,
  author = {Wenting Xu and Viorela Ila and Luping Zhou and Craig T. Jin},
  title = {{TB-HSU: Hierarchical 3D Scene Understanding with Contextual Affordances}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.32969},
}
Figure