Trends and Techniques in 3D Reconstruction and Rendering: A Survey with Emphasis on Gaussian Splatting

Wenhe Chen, Zhengguo Li, Jinxin Guo, Caixia Zheng, Shuilin TianSensors 2025 · DOI

TL;DR

This paper provides a comprehensive survey of 3D reconstruction and rendering techniques, with a particular focus on the recent breakthrough of 3D Gaussian Splatting (3DGS). It highlights how 3DGS offers a powerful, efficient, and high-quality approach to synthesizing novel views of 3D scenes, bridging the gap between traditional methods and neural rendering.

What Problem Does This Paper Address?

The field of 3D reconstruction and rendering is crucial for applications like VR, AR, and autonomous driving. However, traditional methods often struggle with accuracy, computational cost, and handling complex scenes. While deep learning has offered improvements, challenges like data dependency and limited generalization persist. Neural Radiance Fields (NeRFs) have shown promise but often suffer from slow rendering speeds. This survey addresses the need for a consolidated understanding of existing techniques and the emergence of 3D Gaussian Splatting (3DGS) as a highly efficient and effective solution that significantly accelerates real-time rendering and 3D scene estimation.

Methods

The paper categorizes 3D reconstruction methods into passive (using cameras to capture images) and active (using sensors like LiDAR to directly measure depth). Passive methods are further broken down into:

  • Monocular, Binocular, and Multi-camera vision: Discussing techniques like Shape-from-Shading, Photometric Stereo, and Structure from Motion (SFM).
  • Structure from Motion (SFM): Detailed explanation of global, incremental, and hybrid SFM approaches.
  • Multi-view Stereo (MVS): Covering patch-based and depth map merging techniques.

Active methods include:

  • Structured-light-based ranging and laser-based ranging (LiDAR).
  • Time-of-Flight (TOF) sensors.

Deep learning-based reconstruction is categorized by its output representation:

  • Point Cloud-Based: Methods that generate discrete points.
  • Voxel-Based: Using 3D grids to represent scenes.
  • Mesh-Based: Creating polygonal surface models.
  • Depth Map-Based: Reconstructing scenes from depth information.

For rendering, the paper covers:

  • Traditional 3D Rendering: Ray tracing and rasterization.
  • Deep Learning-Based Rendering: Including diffusion models and neural radiance fields (NeRFs).
  • 3D Gaussian Splatting (3DGS): The core focus, which represents scenes using 3D Gaussian functions. This method is highlighted for its efficiency, low memory consumption, and ability to achieve real-time rendering and high-quality reconstruction without neural networks for shape estimation.

Datasets

The paper lists numerous datasets commonly used for 3D reconstruction and rendering tasks. Key datasets mentioned include:

  • TUM RGB-D: Indoor scenes from RGB-D cameras.
  • KITTI: Large-scale outdoor scenes for autonomous driving.
  • SUN3D: Real indoor scenes with RGB-D data.
  • ShapeNet & ModelNet: Large databases of 3D CAD models for recognition and classification.
  • DTU: Multiview dataset with RGB/depth images and point clouds.
  • ETH3D: High-resolution multiview datasets for challenging reconstruction.
  • Tanks and Temples: Multiview reconstruction benchmark.
  • ScanNet: Large-scale RGB-D video dataset of indoor scenes.
  • LLFF: Light field indoor data from multiview smartphone cameras.
  • Mip-NeRF 360: Unbounded scenes for evaluating view synthesis.
  • UT-MM: Multimodal dataset for SLAM and 3D reconstruction.
  • Replica: Large-scale indoor scene reconstruction dataset with dense meshes and HDR textures.
  • BlendedMVS: Synthetic dataset for MVS training.

Key Insights and Ideas

  • 3DGS as a Paradigm Shift: 3D Gaussian Splatting offers a significant leap in rendering speed and quality by directly representing scenes with Gaussian primitives, bypassing the need for complex neural networks for explicit shape estimation and enabling real-time performance.
  • Bridging the Gap: 3DGS effectively combines the strengths of traditional graphics (rasterization) with the expressiveness of implicit representations, offering a more efficient and accessible path to high-fidelity 3D scene reconstruction and rendering.
  • Efficiency and Quality: The paper consistently emphasizes 3DGS's ability to achieve high rendering speeds (FPS) while maintaining excellent image quality (high PSNR, SSIM) and perceptual similarity (low LPIPS), outperforming many NeRF-based methods in speed.
  • Versatility: 3DGS shows promise across various applications, from virtual reality and augmented reality to autonomous driving and medical imaging, due to its adaptability to complex scenes and lighting conditions.
  • Historical Context: The survey provides a valuable historical overview, tracing the evolution of 3D reconstruction and rendering from early geometric methods to modern deep learning approaches, contextualizing 3DGS's impact.

Key Results

The paper highlights the significant performance advantages of 3DGS and its variants in terms of rendering speed. Table 2 shows that 3DGS methods consistently achieve much higher Frames Per Second (FPS) compared to NeRF variants, often by an order of magnitude, while maintaining competitive or superior PSNR and SSIM scores. For instance, on the Mip-NeRF 360 dataset, 3DGS variants achieve FPS in the range of 134-469, whereas NeRF variants are significantly slower. While NeRFs may sometimes achieve slightly higher PSNR or SSIM on specific datasets, 3DGS offers a much better trade-off between speed and quality.

Limitations and Future Work

Limitations:

  • Memory Consumption: 3DGS can still have high memory requirements, especially for large-scale scenes.
  • Scalability: Handling very large and dynamic scenes remains a challenge.
  • Transparent/Reflective Objects: Reconstruction of transparent or highly reflective surfaces can be unstable and may require post-processing.
  • Data Dependency: Like many deep learning methods, 3DGS can be sensitive to the quality and quantity of input data.

Future Work:

  • Hybrid Frameworks: Combining traditional geometric priors with learning-based flexibility for better generalization and interpretability.
  • Lightweight Models: Developing energy-efficient models for deployment on mobile and edge devices.
  • Synthetic Datasets & Self-Supervised Learning: Utilizing large-scale synthetic data and self-supervised learning to reduce reliance on labeled data.
  • Generative Techniques: Employing GANs or diffusion models to improve reconstruction quality.
  • Multi-sensor Fusion: Integrating data from various sensors (e.g., LiDAR, cameras, IMU) for more robust reconstruction.
  • Fine-grained Optimization: Developing algorithms for more precise optimization of geometric details.

Summary updated

Abstract

Three-Dimensional Gaussian Splatting (3DGS), an important advancement in the field of computer graphics and 3D vision, has emerged to greatly accelerate the rendering process in novel views' synthesis. Due to its ability to directly realize the real-time estimation of 3D shapes without neural networks, 3DGS has received a lot of attention in the fields of robotics, urban mapping, autonomous navigation, and virtual reality/augmented reality. In view of the growing popularity of 3DGS, we conduct a systematic review of the relevant literature. We begin by surveying existing work on 3D reconstruction and rendering, outlining the historical development and recent advances from both foundational and innovation-driven perspectives. Next, we summarize the most commonly used datasets and evaluation metrics in 3D reconstruction and rendering. Finally, we summarize the current challenges and suggest potential directions for future research. Through this survey, we aim to provide researchers with a treasure trove of resources in understanding and using techniques related to 3D reconstruction and rendering, in order to promote technological development and application deepening in the field of 3D vision.
BibTeX
@inproceedings{chen2025trends,
  author = {Wenhe Chen and Zhengguo Li and Jinxin Guo and Caixia Zheng and Shuilin Tian},
  title = {{Trends and Techniques in 3D Reconstruction and Rendering: A Survey with Emphasis on Gaussian Splatting}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25123626},
}
Figure