Neural Part Priors: Learning to Optimize Part-Based Object Completion in RGB-D Scans

Alexey Bokhovkin, Angela DaiOpenReview 2022 · OpenReview

Summary

Neural Part Priors (NPPs) introduce a framework for reconstructing complete, part-based 3D objects from noisy and incomplete RGB-D scans. By optimizing learned latent spaces at test-time and enforcing consistency across similar objects within a scene, the method achieves state-of-the-art accuracy in both geometric completion and semantic part segmentation.

The Challenge of Incomplete Scans

While 3D scene understanding has advanced significantly, current methods often treat objects as independent "black boxes." This leads to two primary failures in real-world scenarios:

  1. Incompleteness: Scans from consumer-grade sensors like Kinect are often partial due to occlusions and sensor noise. Standard models struggle to synthesize high-fidelity geometry for missing regions.
  2. Inconsistency: Independent per-object models often reconstruct identical objects (e.g., a set of dining chairs) differently based on their specific partial views, violating the global logic of a coherent scene.

The goal of NPP is to move toward holistic scene reconstruction where objects are decomposed into accurate, complete, and globally consistent parts.

Methodology: Neural Part Priors

The core of the approach is the Neural Part Prior (NPP), which learns a parametric latent space of object parts (e.g., chair legs, seats) represented by coordinate-field MLPs. These MLPs represent geometry as Signed Distance Fields (SDFs), enabling continuous, high-resolution shape representation.

The NPP pipeline: from initial 3D scan and object detection to part-based manifold projection and iterative test-time optimization.

The pipeline consists of four main stages:

  • Detection & Initialization: Objects are detected, and a graph neural network predicts a "semantic recipe" (the parts present and their relationships).
  • Latent Projection: Noisy observations are projected into learned latent spaces for both parts and overall shapes.
  • Test-Time Optimization (TTO): The model iteratively refines latent codes to best fit the observed points, ensuring the reconstruction remains on the manifold of plausible human-made shapes.
  • Scene-Consistent Optimization: Multiple instances of similar objects are optimized jointly, allowing visible parts of one instance to inform the reconstruction of occluded parts in another.

Key Insights: Optimization and Repetition

A fundamental shift in this work is the move from "one-shot prediction" to "test-time optimization." This allows the model to adapt to the specific geometry of a scan while being regularized by strong part-based priors.

NPP leverages scene-level consistency to ensure that repeated objects in a scene are reconstructed with identical, high-fidelity geometry.

By breaking objects into parts, the model utilizes geometric priors for simpler components (like cylinders for legs), which is more robust than learning a prior for every possible complex object design. Furthermore, the "Power of Repetition" is leveraged: human environments are filled with repeated instances, and treating these as a group provides a global constraint that resolves local ambiguities.

Evaluation and Results

NPPs were evaluated on the ScanNet dataset across categories such as chairs, tables, and bookshelves, using PartNet for training and Scan2CAD for ground-truth alignment.

  • Quantitative Performance: NPPs significantly outperformed previous state-of-the-art methods. In Object Completion (measured by Chamfer Distance), NPPs achieved a class average of 0.092, compared to 0.110 for the previous best method.
  • Qualitative Quality: Visual results demonstrate sharp geometries where parts meet perfectly, and repeated objects appear identical, mirroring reality more closely than "blobby" baseline reconstructions.

Limitations and Future Directions

The model currently faces challenges with orientation sensitivity; if the initial rotation estimation is incorrect, optimization may fail. Additionally, while geometric consistency is handled well, the model does not yet account for "stylistic" consistency across different categories (e.g., matching a desk and chair from the same set). Future work may explore equivariant formulations for better rotation handling and higher-level stylistic priors to further enhance scene-wide coherence.

Summary updated

Abstract

3D object recognition has seen significant advances in recent years, showing impressive performance on real-world 3D scan benchmarks, but lacking in object part reasoning, which is fundamental to higher-level scene understanding such as inter-object similarities or object functionality. Thus, we propose to leverage large-scale synthetic datasets of 3D shapes annotated with part information to learn Neural Part Priors (NPPs), optimizable spaces characterizing geometric part priors. Crucially, we can optimize over the learned part priors in order to fit to real-world scanned 3D scenes at test time, enabling robust part decomposition of the real objects in these scenes that also estimates the complete geometry of the object while fitting accurately to the observed real geometry. Moreover, this enables global optimization over geometrically similar detected objects in a scene, which often share strong geometric commonalities, enabling scene-consistent part decompositions. Experiments on the ScanNet dataset demonstrate that NPPs significantly outperforms state of the art in part decomposition and object completion in real-world scenes.
BibTeX
@misc{bokhovkin2022neural,
  author = {Alexey Bokhovkin and Angela Dai},
  title = {{Neural Part Priors: Learning to Optimize Part-Based Object Completion in RGB-D Scans}},
  year = {2022},
  eprint = {2203.09375},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.09375},
  howpublished = {OpenReview},
}
Figure