Instance-Incremental Scene Graph Generation From Real-World Point Clouds via Normalizing Flows

Chao Qi, Jianqin Yin, Jinghang Xu, Pengxiang Ding

TL;DR

Researchers have introduced a new task called instance-incremental scene graph generation, which moves beyond simply describing a 3D scene to actively suggesting how to expand it. By using a novel framework called 3D-ANF (3D Autoregressive Normalizing Flows), the system can take a point cloud of an empty or sparse room and automatically generate a plausible layout of new objects and their spatial relationships, achieving state-of-the-art results on indoor and outdoor benchmarks.

What Problem Does This Paper Address?

Traditional 3D scene graph generation (SGG) is a descriptive task: you give a model a 3D scan, and it tells you what objects are there (nodes) and how they relate to each other (edges), such as "chair standing on floor."

However, applications like Augmented Reality (AR) and interior design robotics need something more proactive. They need to know where a new object should go. This paper addresses the "Instance-incremental" challenge: Given a real-world point cloud (often unstructured and messy), how can we generate a graph that includes both existing elements and logically consistent new instances?

The core difficulty lies in the complexity of real-world layouts. Human environments follow subtle, high-dimensional "rules" (e.g., you don't put a bed in a kitchen, and chairs usually face tables). Modeling these "habits" from raw 3D point clouds is computationally taxing and structurally difficult.

Methods

The authors propose 3D-ANF, a two-stage framework that treats scene expansion as a conditional generation problem.

  1. Graph Representation: First, the model converts the unstructured point cloud into a structured graph. It uses a parameter-sharing Multi-Layer Perceptron (MLP) and max-pooling to extract features from 3D instances (like walls and floors). A Graph Convolutional Network (GCN) then learns the contextual embeddings of this initial scene.
  2. Normalizing Flows (The Core Innovation): To handle the complex distribution of real-world object layouts, the authors use Normalizing Flows. This technique creates an invertible mapping between complex data distributions and a simple Gaussian distribution.
  3. During Training: The model maps real-world "experiences" (observed room layouts) into a Gaussian space.
  4. During Inference: The model samples from this Gaussian space and "reverses" the flow to generate new nodes and edges.
  5. Autoregressive Generation: The process is step-by-step. The model generates one node (object) or edge (relationship) at a time, with each decision conditioned on the previously generated elements.
  6. Constraints: To ensure "common sense," the authors added space constraints (to prevent overcrowding), semantic constraints (to ensure room-appropriate objects), and anti-overlapping checks.

Datasets

The model was evaluated on two main datasets:

  • 3DSSG-O27R16: A standard indoor benchmark derived from 3RScan, containing over 1,300 non-empty rooms with 27 object classes and 16 relationship classes.
  • GPL3D (Graphical-Paris-Lille-3D): A newly introduced dataset by the authors. It adapts the Paris-Lille-3D outdoor urban scan into a graphical format, featuring 15 object classes (like cars and pedestrians) and 9 relationship classes. This helps test if the model generalizes to outdoor environments.

Key Insights and Ideas

  • Generative vs. Descriptive: The shift from "what is here?" to "what should be here?" is a significant step for spatial AI.
  • The Power of Invertibility: By using normalizing flows, the model doesn't just "guess" a layout; it mathematically maps the messy reality of human environments to a clean latent space. This allows for much more stable and diverse generation than standard GANs or VAEs might provide in this context.
  • Context is King: The model doesn't just place a "chair"; it places a "chair" because the GCN identified a "table" and a "floor," and the normalizing flow "remembered" that chairs and tables often share a "standing on" relationship with the floor.
  • Common Sense via Constraints: Purely probabilistic models often produce "hallucinations" (like a chair hanging from a ceiling). The inclusion of explicit semantic and spatial constraints acts as a necessary "sanity check" for the generative process.

Key Results

The 3D-ANF model outperformed existing baselines (like GraphEBM and GraphRNN) across several metrics:

  • Edge Validity: On the 3DSSG dataset, 3D-ANF achieved an edge validity of 84.7%, a massive improvement over the next best baseline (SGG-GEMS at 39.2%). This suggests the model is much better at understanding how objects actually relate to one another.
  • MMD (Maximum Mean Discrepancy): The model achieved the lowest MMD scores (0.44 for degree, 0.08 for cluster), meaning its generated graphs are statistically much closer to real-world human-annotated layouts than other models.
  • Efficiency: Despite its performance, 3D-ANF maintains a relatively low computational footprint, with the fewest FLOPs per scene (1.60G) among the tested generative models.

Limitations and Future Work

While the model is excellent at generating the graph (the "what" and the "how it relates"), it does not yet provide precise 6DoF (6 Degrees of Freedom) pose information—meaning it knows a chair should be near a table, but it doesn't calculate the exact centimeter-perfect coordinates or rotation.

The authors identify object pose prediction as the primary direction for future work. Integrating the graph generation with a geometry-aware pose estimator would allow for a fully automated, end-to-end "room stager" for AR applications.

Summary updated

Abstract

This work introduces a new task of instance-incremental scene graph generation: Given a scene of the point cloud, representing it as a graph and automatically increasing novel instances. A graph denoting the object layout of the scene is finally generated. It is an important task since it helps to guide the insertion of novel 3D objects into a real-world scene in vision-based applications like augmented reality. It is also challenging because the complexity of the real-world point cloud brings difficulties in learning object layout experiences from the observation data (non-empty rooms with labeled semantics). We model this task as a conditional generation problem and propose a 3D autoregressive framework based on normalizing flows (3D-ANF) to address it. First, we represent the point cloud as a graph by extracting the label semantics and contextual relationships. Next, a model based on normalizing flows is introduced to map the conditional generation of graphic elements into the Gaussian process. The mapping is invertible. Thus, the real-world experiences represented in the observation data can be modeled in the training phase, and novel instances can be autoregressively generated based on the Gaussian process in the testing phase. To evaluate the performance of our method sufficiently, we implement this new task on the indoor benchmark dataset 3DSSG-O27R16 and our newly proposed graphical dataset of outdoor scenes GPL3D. Experiments show that our method generates reliable novel graphs from the real-world point cloud and achieves state-of-the-art performance on the datasets.
BibTeX
@article{qi2024instance,
  author = {Chao Qi and Jianqin Yin and Jinghang Xu and Pengxiang Ding},
  title = {{Instance-Incremental Scene Graph Generation From Real-World Point Clouds via Normalizing Flows}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2024},
  doi = {10.1109/tcsvt.2023.3289885},
}
Figure