TL;DR
This paper introduces a novel method for simultaneously refining camera poses and 3D scene geometry represented by decomposed low-rank tensors, using only 2D images. By analyzing the spectral properties of radiance fields and applying Gaussian filtering, their approach significantly improves the robustness and speed of joint optimization, outperforming existing methods.
What Problem Does This Paper Address?
Reconstructing 3D scenes from images, a task central to neural rendering, heavily relies on accurate camera poses. While methods like NeRF have achieved impressive novel view synthesis, they are sensitive to pose inaccuracies. Existing approaches to jointly optimize camera poses and scene representations often struggle with voxel-based methods like TensoRF. These methods, while memory-efficient, lack the inherent spectral biases of MLPs, making them prone to getting stuck in suboptimal solutions during joint optimization, especially when dealing with high-frequency details. This paper aims to bridge this gap by developing a robust and efficient joint optimization framework for decomposed low-rank tensorial radiance fields.
Methods
The core of the proposed method lies in understanding and controlling the spectral properties of the radiance field.
- Spectral Analysis and Gaussian Filtering: The authors conduct a 1D pilot study to demonstrate that naive joint optimization can lead to local optima due to high-frequency signals. They propose using Gaussian filters on both the 2D input images and the 3D radiance field. This filtering acts as a coarse-to-fine strategy, smoothing out high-frequency components initially and allowing for more stable convergence of camera poses and scene geometry.
- Separable Component-Wise Convolution: To efficiently apply Gaussian filtering to the decomposed low-rank tensor representation (like TensoRF), they introduce a "separable component-wise convolution." This technique leverages the structure of the decomposed tensor, applying convolutions to individual components rather than the entire dense tensor, significantly reducing computational overhead.
- Robustness Techniques: To further enhance stability, they incorporate:
- Smoothed 2D Supervision: Blurring input images with scheduled Gaussian kernels.
- Randomly Scaled Kernel Parameters: Using a range of kernel sizes to adapt to varying scene complexities.
- Edge-Guided Loss Mask: Amplifying the loss signal on edge regions of images, as these are crucial for pose estimation.
Datasets
The paper evaluates their method on standard datasets for neural rendering:
- NeRF-Synthetic: A widely used benchmark for novel view synthesis.
- LLFF (Local Light Field Fusion): Another common dataset for novel view synthesis, often used for evaluating pose estimation robustness.
They also perform experiments on planar image alignment tasks, using cropped image patches from a ground truth image.
Key Insights and Ideas
- Spectral Bias is Crucial for Robust Joint Optimization: Unlike MLPs, voxel-based representations (even decomposed ones) lack inherent spectral biases. This makes them susceptible to noise and high-frequency details, hindering robust joint pose optimization.
- Gaussian Filtering as a Coarse-to-Fine Strategy: Applying Gaussian filters to both input images and the 3D scene representation acts as an effective coarse-to-fine mechanism, guiding the optimization towards better solutions.
- Decomposed Tensor Structure Enables Efficient Filtering: The low-rank tensor decomposition allows for a computationally efficient filtering approach (separable component-wise convolution) that avoids reconstructing a dense tensor.
- Robustness Techniques are Essential for Real-World Scenarios: While spectral filtering is key, techniques like edge-guided loss and random kernel scaling are vital for handling the complexities and noise present in real-world datasets.
Key Results
- Improved Novel View Synthesis: The proposed method achieves state-of-the-art novel view synthesis quality on both NeRF-Synthetic and LLFF datasets, often outperforming previous methods in PSNR and LPIPS metrics.
- Reduced Training Time: The method converges significantly faster, requiring only 20-25% of the training iterations compared to previous state-of-the-art methods (e.g., 50k iterations vs. 200k).
- Accurate Pose Estimation: The method demonstrates superior or competitive pose registration accuracy across various scenes and datasets.
- Robustness: Ablation studies confirm the effectiveness of each proposed component (Gaussian filtering, random scaling, edge-guided loss) in improving robustness.
Limitations and Future Work
- Applicability to Other Tensor Decompositions: While the method is primarily demonstrated with TensoRF's VM-decomposition, the authors note its potential applicability to CP-decomposition.
- Computational Cost of Filtering: Although more efficient than brute-force 3D convolution, the filtering still adds computational overhead compared to a non-filtered approach.
- Potential for Further Optimization: The paper suggests that the coarse-to-fine schedule for Gaussian kernel width can be further explored for even better performance.
The authors' work opens avenues for more robust and efficient neural rendering pipelines, particularly in scenarios with unknown or inaccurate camera poses.
Summary updated