Three-dimensional model reconstruction for treasures of jadeite material from uncalibrated image sequences

Chia-Ming Cheng, Shufan Wang, Chin-Hung Teng, Po-Hao Huang, Yu-Chieh Chien, Shang‐Hong LaiProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2005 · DOI

TL;DR

This paper introduces a semi-automatic system for creating detailed 3D models of challenging objects like jadeite from uncalibrated video. It tackles the difficulties of semi-transparent, reflective materials and unknown camera parameters by combining a robust structure-from-motion algorithm with novel image registration and dense depth computation techniques.

What Problem Does This Paper Address?

Reconstructing 3D models of objects from images is a well-studied problem, but it becomes significantly harder when dealing with materials that are both semi-transparent and highly reflective, like jadeite. These properties make it difficult for standard computer vision algorithms to accurately track features and determine camera positions. Furthermore, when the camera's internal parameters (like focal length and lens distortion) and its position and orientation for each image are unknown (uncalibrated), the problem becomes even more complex. The authors aim to overcome these challenges to enable the digital preservation of delicate cultural artifacts like the "Jadeite Cabbage with Insects."

Methods

The proposed system employs a multi-stage approach:

  • Robust Structure from Motion (SfM): This is a core component that recovers camera parameters and a rough 3D structure. The authors highlight a "robust projective reconstruction method" and an "iterative projective to metric upgrading technique" to handle noisy data, missing points due to occlusions, and feature tracking errors. They also introduce a method to merge results from subsets of the image sequence, addressing occlusion issues.
  • Robust Image Registration: This addresses the difficulty of matching features between images due to the material's properties. They propose a novel optical flow computation technique that accounts for brightness variations and uses a gradient-based regularization method to achieve dense matching.
  • Dense Depth Computation: After obtaining camera information and initial structure, this step refines the model by computing dense depth information, integrating color intensity and previous results to create a more detailed structure.
  • Texture Mapping: Finally, a polygon-based greedy algorithm is used to map image textures onto the reconstructed 3D surface.

Datasets

The paper primarily uses the "Jadeite Cabbage with Insects" as a case study for demonstrating their system's performance. They also mention experiments with a "toy house" dataset to showcase the robustness of their method against noise and outliers.

Key Insights and Ideas

  • Handling Specular and Semi-Transparent Materials: The core innovation lies in developing techniques that can cope with the visual ambiguities introduced by jadeite's properties, which standard SfM often struggles with.
  • Robustness to Missing Data and Outliers: The paper emphasizes a robust approach to SfM, specifically designed to handle inevitable missing feature points (due to occlusion) and erroneous feature tracks, which are common in real-world scenarios.
  • Iterative Refinement for Metric Reconstruction: The system iteratively upgrades the projective reconstruction to a metric one, which is crucial for obtaining accurate 3D measurements and poses.
  • Subset-Based Processing for Occlusions: By dividing the image sequence into subsets and merging results, the system can effectively handle occlusions and build a more complete model.

Key Results

The paper demonstrates the effectiveness of their system through experimental results.

  • Jadeite Cabbage: The system successfully reconstructs a 3D model of the "Jadeite Cabbage with Insects," capturing fine details.
  • Toy House: Experiments on the "toy house" dataset show that their robust SfM method effectively eliminates outlier points compared to conventional SfM. Texture mapping is applied to fill the surface, resulting in a rendered model that compares reasonably well to the original.
  • Tolerance to Noise and Missing Data: Figures in the paper illustrate that their refinement step can minimize re-projection errors even with significant amounts of missing data and outliers.

Limitations and Future Work

  • Manual Landmark Selection: The initial step of selecting landmarks is manual due to the material's properties, which is a limitation for full automation.
  • "Open" Surfaces: The reconstructed models are described as "open," meaning occluded regions (like the bottom of an object) cannot be recovered.
  • Future Work: The authors mention plans for further experiments with their cluster-based algorithm for generating final surfaces, suggesting a direction for future research. They also note that while their method handles noise and missing data well, defects can still arise from tracking errors due to a lack of texture information and occlusions.

Summary updated

Abstract

This paper presents several novel techniques in the proposed semi-automatic system for reconstructing three-dimensional models of jadeite material from uncalibrated image sequences. There are two major challenges for this 3D model reconstruction problem. The first is the difficult image registration problem due to the semi-diaphaneity and the highly specular property of jadeite materials. Secondly, the unknown camera information, including the intrinsic (calibration) and extrinsic (position and orientation) parameters, to be recovered from the uncalibrated image sequences makes the robust 3D reconstruction problem very challenging. The proposed 3D modeling process first recovers the camera information as well as rough 3D structure by using a robust structure from motion algorithm, and then further extracts the fine details of the model from dense correspondences between image patches of different views. We have developed three new robust techniques for this challenging task, including a robust structure from motion algorithm, a robust image registration algorithm, and a robust dense depth computation algorithm. Finally, experimental results of the three-dimensional model reconstruction from the image sequence of the Chinese treasure, Jadeite Cabbage with Insects, are given to demonstrate the performance of the developed system.
BibTeX
@inproceedings{cheng2005three,
  author = {Chia-Ming Cheng and Shufan Wang and Chin-Hung Teng and Po-Hao Huang and Yu-Chieh Chien and Shang‐Hong Lai},
  title = {{Three-dimensional model reconstruction for treasures of jadeite material from uncalibrated image sequences}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2005},
  doi = {10.1117/12.587450},
}
Figure