Recent progress in 3D vision

Xiaoxiao Long, Xinjing Cheng, Hao Zhu, Pengju Zhang, Haomin Liu, Jun Li, Lintao Zheng, Qingyong Hu, Xun Cao, Ruigang Yang, Yihong Wu, Guofeng Zhang, Yebin Liu, Kai Xu, Yulan Guo, Baoquan ChenJournal of Image and Graphics 2021 · DOI

TL;DR

This paper provides a comprehensive review of recent advancements in 3D vision, covering key areas like depth estimation, visual localization, 3D modeling, and scene understanding. It highlights the rapid progress driven by deep learning and its applications in fields like autonomous driving and robotics, while also pointing out future research directions.

What Problem Does This Paper Address?

The paper addresses the rapidly growing field of 3D vision, which is crucial for applications like autonomous driving, robotics, digital cities, and virtual/mixed reality. The challenge lies in the vast and diverse research landscape within 3D vision, encompassing tasks from acquiring 3D data to understanding complex scenes. This paper aims to consolidate and review the latest progress in these areas, providing a structured overview of both international and domestic research efforts, and identifying future trends.

Methods

The paper reviews a wide array of methods across several key 3D vision tasks:

  • Depth Image Acquisition: This includes stereo matching (non-end-to-end, end-to-end, and unsupervised approaches) and monocular depth estimation (depth regression and depth completion networks).
  • Visual Localization and Mapping: This covers large-scale visual localization (end-to-end and non-end-to-end methods) and Simultaneous Localization and Mapping (SLAM), including visual SLAM and multi-sensor fusion SLAM.
  • 3D Modeling: This section delves into deep 3D representation learning, deep 3D generative models, structured representation learning, and deep learning-based 3D reconstruction, as well as dynamic human modeling (multi-view RGB, single/multi-depth camera, and single-view RGB methods).
  • 3D Understanding: This focuses on point cloud semantic and instance segmentation.

The paper emphasizes the significant impact of deep learning techniques across all these areas, leading to the development of novel neural network architectures and training strategies.

Datasets

The paper references numerous datasets commonly used in 3D vision research, including:

  • KITTI: A widely used benchmark for stereo matching, visual odometry, and object detection.
  • ScanNet: A dataset for 3D semantic scene understanding.
  • EuRoC: A dataset for visual-inertial SLAM.
  • InStereo2K: A large real dataset for stereo matching in indoor scenes, mentioned as a contribution from domestic research.
  • Flickr1024: A dataset for unsupervised stereo matching, also noted as a domestic contribution.

The paper does not introduce any new datasets.

Key Insights and Ideas

  • Deep Learning as a Driving Force: The paper consistently highlights deep learning as the primary catalyst for recent breakthroughs in 3D vision, enabling more accurate and robust solutions across all surveyed tasks.
  • End-to-End Learning Dominance: For many tasks, particularly stereo matching and visual localization, end-to-end deep learning approaches are increasingly replacing traditional modular pipelines due to their superior performance and integration capabilities.
  • The Rise of Unsupervised and Self-Supervised Methods: Addressing the challenge of data annotation scarcity, unsupervised and self-supervised learning techniques are gaining significant traction, especially in depth estimation and SLAM, demonstrating promising results without requiring extensive labeled data.
  • Fusion is Key for Robustness: For tasks like SLAM, combining information from multiple sensors (e.g., cameras, IMUs, LiDARs) is crucial for achieving robust and reliable performance in challenging environments.
  • Bridging the Gap Between Research and Application: While significant progress has been made in research, the paper implicitly points to the ongoing challenge of deploying these advanced 3D vision techniques into real-world applications, particularly in areas like autonomous driving and robotics, where real-time performance, efficiency, and robustness are paramount.
  • Domestic Progress is Catching Up: The paper acknowledges the growing contributions from Chinese research institutions and companies, particularly in areas like stereo matching datasets and specific SLAM algorithms, indicating a narrowing gap with international leaders.

Key Results

The paper is a review, not an experimental study, so it doesn't present new quantitative results. Instead, it summarizes the state-of-the-art performance achieved by various methods in the surveyed fields. For instance, it notes that:

  • Deep learning-based stereo matching methods have achieved state-of-the-art results on benchmarks like KITTI, often surpassing traditional methods.
  • End-to-end visual localization methods have shown significant improvements in accuracy and robustness.
  • SLAM systems, especially those fusing multiple sensors, are becoming increasingly capable of handling large-scale and complex environments.
  • 3D modeling techniques are enabling more realistic and detailed reconstructions, including dynamic human models.
  • Point cloud segmentation methods are achieving higher accuracy in understanding complex 3D scenes.

Limitations and Future Work

The authors identify several limitations and suggest future research directions:

  • Depth Estimation:
    • Resolution and Accuracy: Improving the resolution and accuracy of depth estimation, especially for stereo matching, remains a challenge.
  • Low-Light Conditions: Robust depth estimation in low-light environments is an open problem.
  • Generalization: Enhancing the generalization capabilities of models to unseen environments and conditions is crucial.
  • Boundary Accuracy: Accurately estimating depth at object boundaries is difficult due to occlusion and smoothing effects.
  • Real-time Performance: Achieving real-time depth estimation with high-resolution outputs requires more efficient network architectures.
  • Visual Localization:
    • Generalization: Improving the generalization of deep learning-based visual localization methods, possibly by incorporating geometric priors.
  • Long-term Robustness: Achieving reliable localization in highly dynamic environments with significant changes over time (e.g., seasons, weather).
  • Efficiency: Developing lightweight models for real-time applications on resource-constrained devices.
  • Challenging Indoor Scenarios: Addressing localization in indoor environments with weak or repetitive textures.
  • SLAM:
    • Scalability and Robustness: Overcoming limitations in scene scale, robustness in complex environments, and power consumption for long-term applications.
  • High-Precision Maps: The need for high-precision 3D maps for large-scale navigation and the development of domestic map-building capabilities.
  • Hardware-Software Integration: The importance of integrating algorithms with hardware for optimized performance.
  • 3D Geometric Modeling:
    • Interactive Modeling: Moving beyond fully automatic generation to intelligent assistance for human modelers.
  • Active 3D Acquisition: Developing data-driven methods for acquiring 3D data of challenging objects (e.g., transparent, reflective).
  • Dataset Development: The need for more large-scale 3D datasets, particularly from domestic sources.
  • Unsupervised/Self-supervised Structured Modeling: Developing methods to learn structured 3D representations without extensive supervision.
  • Reinforcement Learning for Modeling: Exploring RL agents for interactive 3D modeling and editing.
  • Realistic Scene Synthesis: Focusing on generating realistic 3D scenes with plausible object arrangements for applications like interior design and VR.
  • Lightweight Generative Models: Developing efficient models for real-time 3D scene understanding, mapping, and generation for robotics.
  • Dynamic Human Reconstruction:
    • Real-time Performance and Quality: Improving the real-time capabilities and visual fidelity of human reconstruction.
  • Complex Deformations: Handling drastic clothing deformations and topological changes.
  • Single-View Reconstruction: Addressing the significant challenges of self-occlusion and inferring missing information from limited views.
  • Model Compression: Developing efficient methods for compressing 3D models and their associated data.
  • Multi-Person Reconstruction: Advancing techniques for reconstructing multiple interacting humans.
  • Point Cloud Understanding:
    • Large-Scale Segmentation: Developing efficient algorithms for segmenting massive point clouds from sensors like LiDAR.
  • Weakly/Self-Supervised Segmentation: Reducing reliance on dense annotations by leveraging limited or no labels.
  • Temporal Point Cloud Segmentation: Extending segmentation to handle time-series point cloud data.
  • Instance Segmentation: Improving the accuracy and efficiency of distinguishing individual objects within a scene.

Summary updated

Abstract

在自动驾驶、机器人、数字城市以及虚拟/混合现实等应用的驱动下,三维视觉得到了广泛的关注。三维视觉研究主要围绕深度图像获取、视觉定位与制图、三维建模及三维理解等任务而展开。本文围绕上述三维视觉任务,对国内外研究进展进行了综合评述和对比分析。首先,针对深度图像获取任务,从非端到端立体匹配、端到端立体匹配及无监督立体匹配3个方面对立体匹配研究进展进行了回顾,从深度回归网络和深度补全网络两个方面对单目深度估计研究进展进行了回顾。其次,针对视觉定位与制图任务,从端到端视觉定位和非端到端视觉定位两个方面对大场景下的视觉定位研究进展进行了回顾,并从视觉同步定位与地图构建和融合其他传感器的同步定位与地图构建两个方面对同步定位与地图构建的研究进展进行了回顾。再次,针对三维建模任务,从深度三维表征学习、深度三维生成模型、结构化表征学习与生成模型以及基于深度学习的三维重建等4个方面对三维几何建模研究进展进行了回顾,并从多视RGB重建、单深度相机和多深度相机方法以及单视图RGB方法等3个方面对人体动态建模研究进展进行了回顾。最后,针对三维理解任务,从点云语义分割和点云实例分割两个方面对点云语义理解研究进展进行了回顾。在此基础上,给出了三维视觉研究的未来发展趋势,旨在为相关研究者提供参考。
BibTeX
@article{long2021recent,
  author = {Xiaoxiao Long and Xinjing Cheng and Hao Zhu and Pengju Zhang and Haomin Liu and Jun Li and Lintao Zheng and Qingyong Hu and Hao Liu and Xun Cao and Ruigang Yang and Yihong Wu and Guofeng Zhang and Yebin Liu and Kai Xu and Yulan Guo and Baoquan Chen},
  title = {{Recent progress in 3D vision}},
  journal = {Journal of Image and Graphics},
  year = {2021},
  doi = {10.11834/jig.210043},
}
Figure