Overview of image-based 3D reconstruction technology

Yuandong Niu, Limin Liu, Siyuan Huang, Shuangyou ChenJournal of the European Optical Society Rapid Publications 2024 · DOI

TL;DR

This review paper provides a comprehensive roadmap of image-based 3D reconstruction, tracing the evolution from traditional multi-view geometry (MVG) to modern deep learning approaches like MVSNet and NeRF. It synthesizes how these technologies translate 2D image data into various 3D representations—such as point clouds, voxels, and implicit neural fields—essential for the future of autonomous driving, robotics, and digital twins.

What Problem Does This Paper Address?

The fundamental challenge in 3D reconstruction is "recovering the lost dimension": taking 2D images captured by cameras and accurately calculating the depth and structure of the physical world. While traditional methods based on geometric constraints (like Structure from Motion) are mathematically robust, they often produce "holey" models or struggle with untextured surfaces.

Conversely, newer deep learning methods offer the promise of "hole-free" reconstruction and high-fidelity rendering but face hurdles regarding computational cost and generalization to large-scale environments. This paper addresses the need for a unified overview that bridges the gap between classical geometric theory and the current deep learning revolution.

Methods

The authors categorize 3D reconstruction through two primary lenses: Spatial Representation and Algorithmic Approach.

1. Spatial Representation

  • Explicit Representation: Includes Point Clouds (collections of 3D coordinates), Voxels (3D pixels in a grid), and Meshes (vertices and faces). These are intuitive but can be memory-intensive or lack connectivity.
  • Implicit Representation: Uses mathematical functions to describe surfaces. The paper highlights Signed Distance Fields (SDF) and the groundbreaking Neural Radiance Fields (NeRF), which represent scenes as continuous volumes queried by a neural network.

2. Algorithmic Pipelines

  • Traditional Pipeline: Follows a rigid sequence: Feature Extraction (e.g., SIFT, SURF) $\rightarrow$ Stereo Matching (e.g., PatchMatch) $\rightarrow$ Structure from Motion (SfM) $\rightarrow$ Multi-View Stereo (MVS).
  • Deep Learning Pipeline: Replaces or enhances steps with neural networks. For example, MVSNet uses CNNs for cost volume aggregation, while NeRF bypasses the traditional MVS steps entirely to generate a model directly from multi-view images and camera parameters.

Datasets

The paper provides a detailed taxonomy of the "fuel" for these algorithms, categorizing datasets into real-world acquisitions and synthetic models:

  • Autonomous Driving: KITTI, ApolloScape, nuScenes, and BDD100K.
  • Indoor/General Scenes: ScanNet (RGB-D), DTU (multi-view), and Tanks and Temples (large-scale).
  • Synthetic/CAD: ShapeNet (over 3 million models).
  • Specialized: BlendedMVS (for MVS training) and SemanticKITTI (for LiDAR-based semantic understanding).

Key Insights and Ideas

  • Theoretical Continuity: Deep learning isn't a total departure from the past; the feature extraction and stereo matching theories of traditional MVG serve as the essential theoretical foundation for modern neural networks.
  • The "Hole" Problem: A major "aha" moment is the comparison of surface integrity. Traditional MVS filters out low-confidence points to maintain accuracy, which inevitably creates holes. Implicit deep learning methods (like NeRF) solve this by learning a continuous probability distribution of the scene, allowing for seamless surfaces.
  • Representation Matters: The choice between explicit and implicit representation is a trade-off between geometric precision (explicit) and rendering quality/integrity (implicit).

Key Results

  • Performance Shifts: In small-scale indoor scenes, deep learning methods are now outperforming traditional methods in terms of reconstruction accuracy, speed, and surface integrity.
  • Computational Efficiency: The paper notes that GPU-accelerated algorithms (like ADCensus) can achieve up to a 140x speedup over traditional CPU implementations, making real-time reconstruction increasingly viable.
  • Standardization of Evaluation: The authors summarize a rigorous suite of metrics used to judge these models, including PSNR and SSIM for image quality, and Chamfer Distance or IoU (Intersection over Union) for geometric accuracy.

Limitations and Future Work

Despite the rapid progress, the authors identify several "bottlenecks":

  • Generalization: Many deep learning models perform exceptionally well on specific datasets but fail when moved to entirely new environments or object categories.
  • Large-Scale Scaling: While NeRF excels in small scenes, traditional geometric methods still hold the advantage in massive outdoor environments due to the extreme computational and memory demands of training neural fields for large areas.
  • Texture vs. Geometry: Most current evaluation metrics focus on the "shape" (geometry). The authors argue for the development of better indicators to evaluate the quality of reconstructed textures.
  • Real-Time Integration: The ultimate goal is the seamless, real-time integration of virtual and real worlds, which requires further optimization of graphics processor technology and neural network architectures.

Summary updated

Abstract

Three-dimensional (3D) reconstruction technology is the key technology to establish and express the objective world by using computer, and it is widely used in real 3D, automatic driving, aerospace, navigation and industrial robot applications. According to different principles, it is mainly divided into methods based on traditional multi-view geometry and methods based on deep learning. This paper introduces the above methods from the perspective of three-dimensional space representation. The feature extraction and stereo matching theory of traditional 3D reconstruction methods are the theoretical basis of 3D reconstruction methods based on deep learning, so the paper focuses on them. With the development of traditional 3D reconstruction methods and the development of deep learning related theories, the explicit deep learning 3D reconstruction method represented by MVSNet and the implicit 3D reconstruction method represented by NeRF have been gradually developed. At the same time, the dataset and evaluation indicators for 3D reconstruction were introduced. Finally, a summary of image based 3D reconstruction was provided.
BibTeX
@article{niu2024overview,
  author = {Yuandong Niu and Limin Liu and Yuandong Niu and Siyuan Huang and Shuangyou Chen},
  title = {{Overview of image-based 3D reconstruction technology}},
  journal = {Journal of the European Optical Society Rapid Publications},
  year = {2024},
  doi = {10.1051/jeos/2024018},
}
Figure