TL;DR
This paper introduces a semi-automatic system for creating detailed 3D models of challenging objects like jadeite from uncalibrated video. It tackles the difficulties of semi-transparent, reflective materials and unknown camera parameters by combining a robust structure-from-motion algorithm with novel image registration and dense depth computation techniques.
What Problem Does This Paper Address?
Reconstructing 3D models of objects from images is a well-studied problem, but it becomes significantly harder when dealing with materials that are both semi-transparent and highly reflective, like jadeite. These properties make it difficult for standard computer vision algorithms to accurately track features and determine camera positions. Furthermore, when the camera's internal parameters (like focal length and lens distortion) and its position and orientation for each image are unknown (uncalibrated), the problem becomes even more complex. The authors aim to overcome these challenges to enable the digital preservation of delicate cultural artifacts like the "Jadeite Cabbage with Insects."
Methods
The proposed system employs a multi-stage approach:
- Robust Structure from Motion (SfM): This is a core component that recovers camera parameters and a rough 3D structure. The authors highlight a "robust projective reconstruction method" and an "iterative projective to metric upgrading technique" to handle noisy data, missing points due to occlusions, and feature tracking errors. They also introduce a method to merge results from subsets of the image sequence, addressing occlusion issues.
- Robust Image Registration: This addresses the difficulty of matching features between images due to the material's properties. They propose a novel optical flow computation technique that accounts for brightness variations and uses a gradient-based regularization method to achieve dense matching.
- Dense Depth Computation: After obtaining camera information and initial structure, this step refines the model by computing dense depth information, integrating color intensity and previous results to create a more detailed structure.
- Texture Mapping: Finally, a polygon-based greedy algorithm is used to map image textures onto the reconstructed 3D surface.
Datasets
The paper primarily uses the "Jadeite Cabbage with Insects" as a case study for demonstrating their system's performance. They also mention experiments with a "toy house" dataset to showcase the robustness of their method against noise and outliers.
Key Insights and Ideas
- Handling Specular and Semi-Transparent Materials: The core innovation lies in developing techniques that can cope with the visual ambiguities introduced by jadeite's properties, which standard SfM often struggles with.
- Robustness to Missing Data and Outliers: The paper emphasizes a robust approach to SfM, specifically designed to handle inevitable missing feature points (due to occlusion) and erroneous feature tracks, which are common in real-world scenarios.
- Iterative Refinement for Metric Reconstruction: The system iteratively upgrades the projective reconstruction to a metric one, which is crucial for obtaining accurate 3D measurements and poses.
- Subset-Based Processing for Occlusions: By dividing the image sequence into subsets and merging results, the system can effectively handle occlusions and build a more complete model.
Key Results
The paper demonstrates the effectiveness of their system through experimental results.
- Jadeite Cabbage: The system successfully reconstructs a 3D model of the "Jadeite Cabbage with Insects," capturing fine details.
- Toy House: Experiments on the "toy house" dataset show that their robust SfM method effectively eliminates outlier points compared to conventional SfM. Texture mapping is applied to fill the surface, resulting in a rendered model that compares reasonably well to the original.
- Tolerance to Noise and Missing Data: Figures in the paper illustrate that their refinement step can minimize re-projection errors even with significant amounts of missing data and outliers.
Limitations and Future Work
- Manual Landmark Selection: The initial step of selecting landmarks is manual due to the material's properties, which is a limitation for full automation.
- "Open" Surfaces: The reconstructed models are described as "open," meaning occluded regions (like the bottom of an object) cannot be recovered.
- Future Work: The authors mention plans for further experiments with their cluster-based algorithm for generating final surfaces, suggesting a direction for future research. They also note that while their method handles noise and missing data well, defects can still arise from tracking errors due to a lack of texture information and occlusions.
Summary updated