IncR-3DSGG: Incremental Posed RGB Image Based 3D Scene Graph Generation via Privileged Depth Supervision

Mrunmai Phatak, Daniel Kienzle, Roman Küble, Jörg Hähner, Rainer LienhartarXiv 2026 · arXiv

This blog post summarizes the paper "IncR-3DSGG: Incremental Posed RGB Image Based 3D Scene Graph Generation via Privileged Depth Supervision."

TL;DR

IncR-3DSGG is a novel framework that builds metrically grounded 3D scene graphs incrementally from a sequence of posed RGB images, completely eliminating the need for depth sensors at inference time. By leveraging "privileged" depth supervision during training, the system learns to predict 3D spatial relations and distances directly from 2D visual cues, significantly outperforming previous depth-free methods on real-world benchmarks.

What Problem Does This Paper Address?

For an embodied agent (like a robot) to navigate and interact with its environment, it needs more than just a geometric map; it needs a 3D Scene Graph. This is a structured representation that identifies objects and their semantic relationships (e.g., "the cup is on the table").

Most existing 3D scene graph methods rely heavily on depth sensors (like LiDAR or RGB-D cameras) to associate objects across different views and to calculate their 3D positions. This hardware requirement limits deployment on cheaper or smaller platforms. While some "monocular" methods exist, they often struggle with sparse data or inconsistent geometry. The authors tackle the challenge of building these complex 3D representations using only standard RGB images and camera poses (from a standard SLAM system).

Methods

The IncR-3DSGG pipeline operates in two main stages: per-view local graph generation and incremental global merging.

  1. Object Node Detection: For every RGB frame, a frozen Encoder-Only Mask Transformer (EoMT) performs panoptic segmentation to identify object instances and extract visual features.
  2. Parametric Relation Predictor (PRP): This is the core innovation. The PRP takes 2D visual features and 2D geometric cues (like bounding box overlap and relative scale) and jointly predicts:
  3. Semantic Relations: Labels like supported by, standing on, or attached to.
  4. 3D Geometry: Pairwise 3D measurements including centroid distance, azimuth, and elevation.
  5. Privileged Supervision: Crucially, the PRP is trained using 3D ground truth (depth/point clouds), but at test time, it works purely from 2D inputs.
  6. Incremental Merging & PGO: As the agent moves, new detections are matched to existing global nodes using Hungarian matching. The noisy 3D geometric predictions from the PRP are then fused across views using Pose Graph Optimization (PGO). This step treats objects as landmarks and optimizes their positions to be globally consistent.
Overview of the IncR-3DSGG framework showing the flow from RGB images to a global scene graph.

Datasets

The authors evaluated their method on two primary platforms:

  • 3RScan / 3DSSG: A large-scale dataset of real-world indoor scans with expert-annotated 3D scene graphs. They specifically used the "SPARSE" setting, which is the most challenging as it relies on RGB-derived geometry.
  • AI2-THOR: An interactive simulator used for controlled experiments, such as testing robustness against camera rotation and noise-free ablation studies.

Key Insights and Ideas

  • 3D Reasoning from 2D Cues: The paper demonstrates that 3D geometric reasoning is essential for scene graphs, but the 3D signal itself (depth) isn't strictly required at deployment. A model can "learn" the 3D implications of 2D visual patterns if supervised correctly during training.
  • Relational Localization: Unlike standard SLAM which localizes objects independently, IncR-3DSGG propagates spatial information through the graph. Well-observed "landmark" objects help improve the localization of smaller or rarely seen objects through their predicted relationships.
  • Contextualized Embeddings: The PRP uses a Transformer encoder (see Figure 3) to allow objects in a view to "attend" to one another. This means when the model reasons about a "cup," its representation already encodes the presence of a "table" nearby, leading to more accurate relation predictions.
The Parametric Relation Predictor (PRP) architecture.

Key Results

On the 3RScan benchmark, IncR-3DSGG achieved state-of-the-art results for depth-free methods:

  • Object Recall: Reached 52.2 mean recall, nearly doubling the performance of the previous best method, MonoSSG (27.0).
  • Predicate Recall: Attained 30.8 mean recall compared to MonoSSG’s 17.6.
  • Efficiency: The relational components (PRP, association, and PGO) are highly efficient, adding less than 2% overhead (approx. 16ms) to the standard segmentation front-end.

Qualitatively, the system builds a coherent map over time, refining object positions as more views are processed.

Figure 1. Incremental construction of the global scene graph GG over 20 processed views. Nodes are instantiated when an object is first detected and refined as later views contribute pairwise con- straints. The scan is drawn in the global coordinate frame purely as a backdrop, showing how our predicted nodes align with the annotated instances. Our method observes no depth and builds no point cloud or mesh, producing object positions and relations di- rectly.

Limitations and Future Work

  • Static Environments: The current system assumes the world is static. Handling dynamic objects (people moving, furniture being shifted) remains a challenge for future work.
  • Pose Dependency: The method relies on accurate camera extrinsics from an off-the-shelf SLAM system. Significant drift in SLAM would likely degrade the global graph consistency.
  • Instance Coverage: The authors note that "triplet recall" (correctly identifying subject, object, and relation) is currently bottlenecked by the 2D detector's ability to find all annotated objects in every frame.
  • Active Exploration: A natural next step is coupling this incremental builder with an exploration policy that actively chooses camera views to reduce uncertainty in the scene graph.

Summary updated

Abstract

Scene graphs provide a compact, structured representation of environments by capturing objects and their spatial or semantic relationships in a form suitable for embodied agents. Existing 3D scene graph generation methods often rely on depth sensing at inference, either to associate object instances across views via point cloud overlap or to compute inter-object relations from geometric cues, limiting deployment to platforms with depth sensors. We propose IncR-3DSGG, which builds 3D scene graphs incrementally from posed\footnote{RGB images with camera extrinsics estimated by an off-the-shelf visual SLAM system.} RGB images, eliminating the use of depth sensors at inference. Our method leverages a Parametric Relation Predictor (PRP) that learns to estimate pairwise distances and relative angles between objects from 2D visual features and 2D geometric cues, using 3D geometric supervision available only during training. These learned pairwise measurements are fused across views through Pose Graph Optimization (PGO)~\cite{grisetti2010tutorial} to recover metrically grounded 3D object positions and build a globally consistent 3D scene graph incrementally. On the real-world 3RScan~\cite{wald2019rio} dataset, our method attains $52.2$ mean object recall and $30.8$ mean predicate recall under the 3DSSG~\cite{wald2020learning} evaluation protocol, against $27.0$ and $17.6$ for MonoSSG~\cite{wu2023incremental}, while using no depth or reconstructed geometry at inference. This shows that relations and metrically grounded 3D object representations can be predicted from posed RGB images alone.
BibTeX
@misc{phatak2026incr,
  author = {Mrunmai Phatak and Daniel Kienzle and Roman Küble and Jörg Hähner and Rainer Lienhart},
  title = {{IncR-3DSGG: Incremental Posed RGB Image Based 3D Scene Graph Generation via Privileged Depth Supervision}},
  year = {2026},
  eprint = {2611.15429},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15429},
}
Figure