This blog post summarizes the paper "IncR-3DSGG: Incremental Posed RGB Image Based 3D Scene Graph Generation via Privileged Depth Supervision."
TL;DR
IncR-3DSGG is a novel framework that builds metrically grounded 3D scene graphs incrementally from a sequence of posed RGB images, completely eliminating the need for depth sensors at inference time. By leveraging "privileged" depth supervision during training, the system learns to predict 3D spatial relations and distances directly from 2D visual cues, significantly outperforming previous depth-free methods on real-world benchmarks.
What Problem Does This Paper Address?
For an embodied agent (like a robot) to navigate and interact with its environment, it needs more than just a geometric map; it needs a 3D Scene Graph. This is a structured representation that identifies objects and their semantic relationships (e.g., "the cup is on the table").
Most existing 3D scene graph methods rely heavily on depth sensors (like LiDAR or RGB-D cameras) to associate objects across different views and to calculate their 3D positions. This hardware requirement limits deployment on cheaper or smaller platforms. While some "monocular" methods exist, they often struggle with sparse data or inconsistent geometry. The authors tackle the challenge of building these complex 3D representations using only standard RGB images and camera poses (from a standard SLAM system).
Methods
The IncR-3DSGG pipeline operates in two main stages: per-view local graph generation and incremental global merging.
- Object Node Detection: For every RGB frame, a frozen Encoder-Only Mask Transformer (EoMT) performs panoptic segmentation to identify object instances and extract visual features.
- Parametric Relation Predictor (PRP): This is the core innovation. The PRP takes 2D visual features and 2D geometric cues (like bounding box overlap and relative scale) and jointly predicts:
- Semantic Relations: Labels like supported by, standing on, or attached to.
- 3D Geometry: Pairwise 3D measurements including centroid distance, azimuth, and elevation.
- Privileged Supervision: Crucially, the PRP is trained using 3D ground truth (depth/point clouds), but at test time, it works purely from 2D inputs.
- Incremental Merging & PGO: As the agent moves, new detections are matched to existing global nodes using Hungarian matching. The noisy 3D geometric predictions from the PRP are then fused across views using Pose Graph Optimization (PGO). This step treats objects as landmarks and optimizes their positions to be globally consistent.
Datasets
The authors evaluated their method on two primary platforms:
- 3RScan / 3DSSG: A large-scale dataset of real-world indoor scans with expert-annotated 3D scene graphs. They specifically used the "SPARSE" setting, which is the most challenging as it relies on RGB-derived geometry.
- AI2-THOR: An interactive simulator used for controlled experiments, such as testing robustness against camera rotation and noise-free ablation studies.
Key Insights and Ideas
- 3D Reasoning from 2D Cues: The paper demonstrates that 3D geometric reasoning is essential for scene graphs, but the 3D signal itself (depth) isn't strictly required at deployment. A model can "learn" the 3D implications of 2D visual patterns if supervised correctly during training.
- Relational Localization: Unlike standard SLAM which localizes objects independently, IncR-3DSGG propagates spatial information through the graph. Well-observed "landmark" objects help improve the localization of smaller or rarely seen objects through their predicted relationships.
- Contextualized Embeddings: The PRP uses a Transformer encoder (see Figure 3) to allow objects in a view to "attend" to one another. This means when the model reasons about a "cup," its representation already encodes the presence of a "table" nearby, leading to more accurate relation predictions.
Key Results
On the 3RScan benchmark, IncR-3DSGG achieved state-of-the-art results for depth-free methods:
- Object Recall: Reached 52.2 mean recall, nearly doubling the performance of the previous best method, MonoSSG (27.0).
- Predicate Recall: Attained 30.8 mean recall compared to MonoSSG’s 17.6.
- Efficiency: The relational components (PRP, association, and PGO) are highly efficient, adding less than 2% overhead (approx. 16ms) to the standard segmentation front-end.
Qualitatively, the system builds a coherent map over time, refining object positions as more views are processed.
Limitations and Future Work
- Static Environments: The current system assumes the world is static. Handling dynamic objects (people moving, furniture being shifted) remains a challenge for future work.
- Pose Dependency: The method relies on accurate camera extrinsics from an off-the-shelf SLAM system. Significant drift in SLAM would likely degrade the global graph consistency.
- Instance Coverage: The authors note that "triplet recall" (correctly identifying subject, object, and relation) is currently bottlenecked by the 2D detector's ability to find all annotated objects in every frame.
- Active Exploration: A natural next step is coupling this incremental builder with an exploration policy that actively chooses camera views to reduce uncertainty in the scene graph.
Summary updated