Abstract
Traditional 3D scene graph generation (SGG) focuses on describing existing scenes. However, emerging applications in Augmented Reality (AR) and robotics require models that can suggest logical expansions to sparse environments. We present 3D-ANF (3D Autoregressive Normalizing Flows), a framework for instance-incremental scene graph generation. By mapping complex real-world object distributions to a latent Gaussian space using normalizing flows, 3D-ANF generates plausible new objects and relationships conditioned on existing point cloud data. Our approach achieves state-of-the-art performance on indoor (3DSSG) and outdoor (GPL3D) benchmarks, significantly improving edge validity and statistical alignment with human-annotated layouts.
Introduction: The Instance-incremental Challenge
Traditional 3D scene graph generation (SGG) is primarily a descriptive task. Given a 3D scan, a model identifies objects (nodes) and their spatial relationships (edges), such as a "chair standing on a floor." While useful for scene understanding, this approach is reactive. Proactive applications, such as interior design assistants or AR-based room staging, require the ability to predict what should be in a scene.
This research introduces the instance-incremental scene graph generation task. Unlike traditional SGG, this task requires the model to take a real-world point cloud of a sparse or empty room and generate a graph that includes both existing elements and logically consistent new instances.
The core challenge lies in the high-dimensional, implicit rules governing human environments. For instance, furniture placement is constrained by room type, functional proximity, and spatial availability. Modeling these "habits" from unstructured and often noisy 3D point clouds is computationally intensive and structurally complex.
Methodology: 3D-ANF
The proposed framework, 3D-ANF, treats scene expansion as a conditional generation problem using a two-stage process.
Graph Representation
First, the model extracts features from the unstructured point cloud. It employs a parameter-sharing Multi-Layer Perceptron (MLP) and max-pooling to derive initial instance features (e.g., for walls and floors). These are then processed by a Graph Convolutional Network (GCN) to learn contextual embeddings that represent the initial scene state.
Normalizing Flows and Autoregressive Generation
The heart of the system is the Normalizing Flows module. Normalizing flows provide an invertible mapping between complex data distributions and a simple Gaussian distribution. During training, the model learns to map observed real-world layouts into this Gaussian space. During inference, the model samples from the Gaussian distribution and "reverses" the flow to generate new nodes and edges.
The generation process is autoregressive, meaning the model produces one node or edge at a time. Each decision is conditioned on the previously generated graph structure. To maintain physical and semantic plausibility, the authors incorporate several constraints:
- Space Constraints: Preventing overcrowding in the 3D volume.
- Semantic Constraints: Ensuring objects are appropriate for the room type.
- Anti-overlapping Checks: Verifying that new instances do not occupy the same physical space as existing ones.
Experimental Evaluation
The model was evaluated on two benchmarks:
- 3DSSG-O27R16: An indoor dataset derived from 3RScan, featuring 27 object classes and 16 relationship types across 1,300+ rooms.
- GPL3D (Graphical-Paris-Lille-3D): A new outdoor dataset introduced in this work, adapting urban scans into a graphical format with 15 object classes and 9 relationship types.
3D-ANF significantly outperformed baselines like GraphEBM and GraphRNN. Notably, on the 3DSSG dataset, it achieved an edge validity of 84.7%, compared to 39.2% for the best baseline. Statistical metrics like Maximum Mean Discrepancy (MMD) also showed that 3D-ANF's generated graphs are much closer to human-annotated ground truth. Furthermore, the model is computationally efficient, requiring only 1.60G FLOPs per scene.
Conclusion
3D-ANF represents a significant shift from descriptive to generative spatial AI. By leveraging the invertibility of normalizing flows, the model captures the nuances of human environments more effectively than previous generative approaches. While the current model excels at graph generation, it does not yet predict precise 6DoF poses (exact coordinates and rotations). Future work aims to integrate geometry-aware pose estimation to enable end-to-end automated room staging.
Summary updated