TL;DR
This paper introduces IVGM, the first large-scale indoor dataset with comprehensive visual and geometric ground truth, designed to push the boundaries of novel view synthesis (NVS) in complex environments. They also present a benchmark and evaluate several NVS methods, demonstrating the critical role of high-quality geometric data and multi-view inputs for realistic scene reconstruction.
What Problem Does This Paper Address?
Accurately reconstructing indoor spaces and generating novel views (NVS) is crucial for AR, VR, and robotics. However, existing indoor datasets are often limited in scale, lack dense 3D ground truth (like point clouds), and have insufficient viewpoints, hindering the development of robust NVS techniques for real-world applications. Traditional methods like Structure-from-Motion (SfM) struggle with repetitive textures and feature-scarce environments common indoors, leading to inaccurate camera poses and reconstructions. While neural methods like NeRF have advanced NVS, they still face challenges with large-scale scenes and require high-quality input data.
Methods
The authors developed a custom data collection platform mounted on an electric Go-Kart, equipped with:
- Ouster OS0-128 LiDAR: For high-density, wide-field-of-view point clouds.
- Insta360 ONE RS 1-Inch 360 Edition & Insta360 Titan cameras: To capture high-resolution panoramic imagery.
- Xsens Mti-630 IMU: For sensor fusion and odometry.
They employed a LiDAR-visual-inertial SLAM algorithm for precise camera pose estimation, especially in challenging areas. For data processing, they developed a pipeline to undistort fisheye images into pinhole models, creating single-view and five-view datasets. They also downsampled LiDAR point clouds for efficient processing.
Datasets
- IVGM Dataset: The core contribution, featuring three large-scale indoor scenes: two office areas and an underground garage. It provides:
- High-resolution panoramic images.
- Dense LiDAR point clouds.
- Textured meshes.
- Accurate camera poses.
- Existing Datasets (for comparison): ScanNet, ScanNet++, ARKitScenes, MuSHRoom (indoor); Tanks and Temples, GigaMVS, KITTI-360 (large-scale, mostly outdoor). The paper highlights the limitations of these existing datasets for large-scale indoor NVS.
Key Insights and Ideas
- Multi-view is Crucial: Synthesizing views from unobserved angles (free-view synthesis) is significantly impacted by the quality and quantity of training views. Using multiple views (like the five-view setup) dramatically improves rendering quality compared to single-view training.
- Geometric Priors Matter: Integrating high-quality point clouds (especially dense LiDAR data) significantly enhances NVS performance (higher PSNR/SSIM, lower LPIPS) compared to using only images or sparse SfM point clouds.
- Camera Quality Impacts Performance: Higher-resolution and wider-field-of-view cameras (like the Insta360 Titan) generally lead to better NVS results, though exposure bias can be an issue.
- Gaussian Splatting Dominates: 3D Gaussian Splatting consistently outperformed other tested NVS methods across various scenes and input types, demonstrating its effectiveness for high-quality rendering in complex indoor environments.
Key Results
- Quantitative Benchmark: The paper presents quantitative results (PSNR, SSIM, LPIPS) for several NVS algorithms (Instant-NGP, Nerfacto, Nerfacto-huge, READ, Gaussian Splatting) on the IVGM dataset.
- Gaussian Splatting achieved the highest scores across most metrics and scenes, often with PSNR > 30 and SSIM > 0.9.
- The five-view setup consistently improved performance over single-view inputs for most methods.
- LiDAR point clouds (especially denser ones) significantly boosted Gaussian Splatting's performance compared to SfM point clouds.
- Qualitative Results: Figure 4 and Figure 6 visually demonstrate the superior rendering quality of Gaussian Splatting and the benefits of multi-view inputs and LiDAR data.
Limitations and Future Work
- Exposure Bias: The industrial-grade Titan camera, while high-quality, exhibits exposure bias, leading to overexposure issues. Future work could focus on harmonizing exposure and color profiles from multiple cameras.
- RAW Data: The panoramic cameras used do not support RAW format, limiting the ability to capture images under varying lighting conditions and perform advanced post-processing. Future work plans to upgrade cameras for fixed video recordings to facilitate multiple acquisitions.
- LiDAR Utilization: While Gaussian Splatting benefits from LiDAR, it may not fully utilize the high-quality point clouds for initialization. Better utilization could further enhance rendering quality.
- Dataset Expansion: The authors plan to continuously update the dataset by capturing more extensive and larger scenes.
- Training Challenges: Training on large scenes is computationally intensive. Future work may explore distributed training systems or improved segmentation techniques.
Summary updated