Instance-incremental Scene Graph Generation from Real-world Point Clouds via Normalizing Flows

Chao Qi, Jianqin Yin, Jinghang Xu, Pengxiang Ding

Abstract

Traditional 3D scene graph generation (SGG) focuses on describing existing scenes. However, emerging applications in Augmented Reality (AR) and robotics require models that can suggest logical expansions to sparse environments. We present 3D-ANF (3D Autoregressive Normalizing Flows), a framework for instance-incremental scene graph generation. By mapping complex real-world object distributions to a latent Gaussian space using normalizing flows, 3D-ANF generates plausible new objects and relationships conditioned on existing point cloud data. Our approach achieves state-of-the-art performance on indoor (3DSSG) and outdoor (GPL3D) benchmarks, significantly improving edge validity and statistical alignment with human-annotated layouts.

Introduction: The Instance-incremental Challenge

Traditional 3D scene graph generation (SGG) is primarily a descriptive task. Given a 3D scan, a model identifies objects (nodes) and their spatial relationships (edges), such as a "chair standing on a floor." While useful for scene understanding, this approach is reactive. Proactive applications, such as interior design assistants or AR-based room staging, require the ability to predict what should be in a scene.

This research introduces the instance-incremental scene graph generation task. Unlike traditional SGG, this task requires the model to take a real-world point cloud of a sparse or empty room and generate a graph that includes both existing elements and logically consistent new instances.

Comparison between traditional SGG and the proposed instance-incremental SGG task.

The core challenge lies in the high-dimensional, implicit rules governing human environments. For instance, furniture placement is constrained by room type, functional proximity, and spatial availability. Modeling these "habits" from unstructured and often noisy 3D point clouds is computationally intensive and structurally complex.

Methodology: 3D-ANF

The proposed framework, 3D-ANF, treats scene expansion as a conditional generation problem using a two-stage process.

Graph Representation

First, the model extracts features from the unstructured point cloud. It employs a parameter-sharing Multi-Layer Perceptron (MLP) and max-pooling to derive initial instance features (e.g., for walls and floors). These are then processed by a Graph Convolutional Network (GCN) to learn contextual embeddings that represent the initial scene state.

Normalizing Flows and Autoregressive Generation

The heart of the system is the Normalizing Flows module. Normalizing flows provide an invertible mapping between complex data distributions and a simple Gaussian distribution. During training, the model learns to map observed real-world layouts into this Gaussian space. During inference, the model samples from the Gaussian distribution and "reverses" the flow to generate new nodes and edges.

The 3D-ANF framework, illustrating the training process (distribution mapping) and the autoregressive inference process.

The generation process is autoregressive, meaning the model produces one node or edge at a time. Each decision is conditioned on the previously generated graph structure. To maintain physical and semantic plausibility, the authors incorporate several constraints:

  • Space Constraints: Preventing overcrowding in the 3D volume.
  • Semantic Constraints: Ensuring objects are appropriate for the room type.
  • Anti-overlapping Checks: Verifying that new instances do not occupy the same physical space as existing ones.

Experimental Evaluation

The model was evaluated on two benchmarks:

  1. 3DSSG-O27R16: An indoor dataset derived from 3RScan, featuring 27 object classes and 16 relationship types across 1,300+ rooms.
  2. GPL3D (Graphical-Paris-Lille-3D): A new outdoor dataset introduced in this work, adapting urban scans into a graphical format with 15 object classes and 9 relationship types.

3D-ANF significantly outperformed baselines like GraphEBM and GraphRNN. Notably, on the 3DSSG dataset, it achieved an edge validity of 84.7%, compared to 39.2% for the best baseline. Statistical metrics like Maximum Mean Discrepancy (MMD) also showed that 3D-ANF's generated graphs are much closer to human-annotated ground truth. Furthermore, the model is computationally efficient, requiring only 1.60G FLOPs per scene.

Conclusion

3D-ANF represents a significant shift from descriptive to generative spatial AI. By leveraging the invertibility of normalizing flows, the model captures the nuances of human environments more effectively than previous generative approaches. While the current model excels at graph generation, it does not yet predict precise 6DoF poses (exact coordinates and rotations). Future work aims to integrate geometry-aware pose estimation to enable end-to-end automated room staging.

Summary updated

BibTeX
@article{qi2024instance,
  author = {Chao Qi and Jianqin Yin and Jinghang Xu and Pengxiang Ding},
  title = {{Instance-incremental Scene Graph Generation from Real-world Point Clouds via Normalizing Flows}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2024},
  doi = {10.1109/tcsvt.2023.3289885},
}
Figure