Summary
Neural Part Priors (NPPs) introduce a framework for reconstructing complete, part-based 3D objects from noisy and incomplete RGB-D scans. By optimizing learned latent spaces at test-time and enforcing consistency across similar objects within a scene, the method achieves state-of-the-art accuracy in both geometric completion and semantic part segmentation.
The Challenge of Incomplete Scans
While 3D scene understanding has advanced significantly, current methods often treat objects as independent "black boxes." This leads to two primary failures in real-world scenarios:
- Incompleteness: Scans from consumer-grade sensors like Kinect are often partial due to occlusions and sensor noise. Standard models struggle to synthesize high-fidelity geometry for missing regions.
- Inconsistency: Independent per-object models often reconstruct identical objects (e.g., a set of dining chairs) differently based on their specific partial views, violating the global logic of a coherent scene.
The goal of NPP is to move toward holistic scene reconstruction where objects are decomposed into accurate, complete, and globally consistent parts.
Methodology: Neural Part Priors
The core of the approach is the Neural Part Prior (NPP), which learns a parametric latent space of object parts (e.g., chair legs, seats) represented by coordinate-field MLPs. These MLPs represent geometry as Signed Distance Fields (SDFs), enabling continuous, high-resolution shape representation.
The pipeline consists of four main stages:
- Detection & Initialization: Objects are detected, and a graph neural network predicts a "semantic recipe" (the parts present and their relationships).
- Latent Projection: Noisy observations are projected into learned latent spaces for both parts and overall shapes.
- Test-Time Optimization (TTO): The model iteratively refines latent codes to best fit the observed points, ensuring the reconstruction remains on the manifold of plausible human-made shapes.
- Scene-Consistent Optimization: Multiple instances of similar objects are optimized jointly, allowing visible parts of one instance to inform the reconstruction of occluded parts in another.
Key Insights: Optimization and Repetition
A fundamental shift in this work is the move from "one-shot prediction" to "test-time optimization." This allows the model to adapt to the specific geometry of a scan while being regularized by strong part-based priors.
By breaking objects into parts, the model utilizes geometric priors for simpler components (like cylinders for legs), which is more robust than learning a prior for every possible complex object design. Furthermore, the "Power of Repetition" is leveraged: human environments are filled with repeated instances, and treating these as a group provides a global constraint that resolves local ambiguities.
Evaluation and Results
NPPs were evaluated on the ScanNet dataset across categories such as chairs, tables, and bookshelves, using PartNet for training and Scan2CAD for ground-truth alignment.
- Quantitative Performance: NPPs significantly outperformed previous state-of-the-art methods. In Object Completion (measured by Chamfer Distance), NPPs achieved a class average of 0.092, compared to 0.110 for the previous best method.
- Qualitative Quality: Visual results demonstrate sharp geometries where parts meet perfectly, and repeated objects appear identical, mirroring reality more closely than "blobby" baseline reconstructions.
Limitations and Future Directions
The model currently faces challenges with orientation sensitivity; if the initial rotation estimation is incorrect, optimization may fail. Additionally, while geometric consistency is handled well, the model does not yet account for "stylistic" consistency across different categories (e.g., matching a desk and chair from the same set). Future work may explore equivariant formulations for better rotation handling and higher-level stylistic priors to further enhance scene-wide coherence.
Summary updated