TL;DR
This paper introduces a novel approach for reconstructing detailed 3D models of man-made objects from noisy, incomplete single-view scans (like those from a Kinect). It achieves this by intelligently assembling pre-existing 3D parts from a repository, effectively filling in missing geometry and recovering the object's underlying structure.
What Problem Does This Paper Address?
Acquiring high-quality 3D models of real-world objects is challenging, especially with consumer-grade scanners like the Kinect. These devices often produce scans that are noisy, incomplete, and contain missing parts due to occlusions or material properties. While existing methods focus on geometric completion, they struggle to recover the underlying semantic structure (e.g., identifying the seat, legs, and back of a chair). Furthermore, template-based methods require globally similar templates, which are scarce in large repositories. This paper tackles the problem of recovering both the structure and semantic labels of man-made objects from such low-quality, single-view scans, even when significant portions of the object are missing.
Methods
The core of the proposed method is a "bottom-up" part assembly approach. It involves three main stages:
- Candidate Parts Selection: To efficiently find relevant parts, the system first roughly aligns each repository model with the input scan. Then, it searches for candidate parts within a small 3D window around the expected location of that part in the input scan. This selection is guided by a novel scoring function that considers both the geometric fidelity (how well a part fits the 3D scan) and the geometric contribution (how much of the input scan and image contour the part covers). This step leverages both the 3D scan and the corresponding RGB image to overcome missing data.
- Structure Composition: This stage intelligently combines the selected candidate parts to form a coherent structure. It uses a backtracking search algorithm that considers geometric constraints such as proximity (parts shouldn't be too far apart), overlap (parts shouldn't intersect excessively), and geometric fidelity. The system also incorporates symmetry detection to automatically add symmetric counterparts of parts.
- Part Conjoining: The final step refines the assembled parts into a well-connected and visually pleasing model. It identifies contact points between parts and then optimizes their sizes and positions using a global optimization process to ensure they meet correctly. This stage uses both primary contact points (identified from repository models) and secondary contact points (generated based on proximity) to achieve a robust connection.
Datasets
The paper uses several datasets of man-made objects for training and evaluation:
- Chair Dataset: Contains 70 chair models.
- Table Dataset: Contains 61 table models.
- Bicycle Dataset: Contains 38 bicycle models.
- Airplane Dataset: Contains 70 airplane models.
All models in these datasets are pre-segmented into semantically labeled parts. The authors do not mention introducing new benchmarks but rather use existing models to demonstrate their approach.
Key Insights and Ideas
- Leveraging RGB Image for Structure Recovery: The paper highlights the crucial role of the RGB image, especially for recovering parts that are severely missing or have scanner-unfriendly materials (like reflective surfaces) in the 3D scan.
- Local-to-Global Assembly: Unlike global template-matching methods, this approach builds the structure from local part matches in a bottom-up fashion, making it more robust to incomplete data and the scarcity of perfectly matching templates.
- Part-based Scoring for Candidate Selection: The novel scoring function that balances geometric fidelity and contribution is key to efficiently filtering relevant parts from a large repository.
- Importance of Secondary Contact Points: The introduction of secondary contact points in the conjoining stage provides more flexibility for connecting parts, especially in complex scenarios.
- Low-Dimensional Shape Space Assumption: The method relies on the observation that many man-made object classes lie in a low-dimensional shape space defined by the relative positions and sizes of their parts, which enables efficient search and composition.
Key Results
The paper demonstrates successful structure recovery for various man-made objects (chairs, tables, bicycles, airplanes) from single-view, noisy, and incomplete Kinect scans. The results show that the approach can faithfully recover underlying structures and semantic parts, even when significant portions of the object are missing. The method is largely automatic, requiring minimal user interaction for initial object segmentation. The part selection step takes about a minute, and structure composition takes about two minutes on a standard computer, with part conjoining being very fast (<1 second). The results show that the quality of reconstruction is directly related to the richness of the repository, with larger repositories yielding more faithful results.
Limitations and Future Work
The authors acknowledge several limitations:
- Assumption of Low-Dimensional Shape Space: The method may struggle with objects whose shape variations cannot be characterized by the spatial layout of parts (e.g., facades with irregular structures).
- Structure Recovery vs. Fine Geometry: The focus is on structure recovery, not fine geometric reconstruction. While the assembled models are geometrically similar, they may exhibit noticeable shape differences, which could be addressed by existing non-rigid deformation techniques.
- Resolution Limitations: The low resolution of input scans can make it difficult to discriminate between closely spaced parts, potentially leading to incorrect matches.
- Dependency on Repository: The approach is limited by the availability of suitable parts in the repository. Uncommon structures or highly curved geometries may be poorly recovered if corresponding parts are missing.
- Future Work: The authors suggest extending the technique to multi-view inputs to reduce user interaction and improve robustness. They also propose investigating the interdependence of structure composition and candidate selection steps, and incorporating style or functional constraints for more coherent models. Finally, they aim to extend the approach to reconstruct entire indoor scenes.
Summary updated