Improving Robustness for Joint Optimization of Camera Pose and Decomposed Low-Rank Tensorial Radiance Fields

Bo-Yu Chen, Wei-Chen Chiu, Yu-Lun LiuAAAI 2024 · OpenReview

TL;DR

This paper introduces a novel method for simultaneously refining camera poses and 3D scene geometry represented by decomposed low-rank tensors, using only 2D images. By analyzing the spectral properties of radiance fields and applying Gaussian filtering, their approach significantly improves the robustness and speed of joint optimization, outperforming existing methods.

What Problem Does This Paper Address?

Reconstructing 3D scenes from images, a task central to neural rendering, heavily relies on accurate camera poses. While methods like NeRF have achieved impressive novel view synthesis, they are sensitive to pose inaccuracies. Existing approaches to jointly optimize camera poses and scene representations often struggle with voxel-based methods like TensoRF. These methods, while memory-efficient, lack the inherent spectral biases of MLPs, making them prone to getting stuck in suboptimal solutions during joint optimization, especially when dealing with high-frequency details. This paper aims to bridge this gap by developing a robust and efficient joint optimization framework for decomposed low-rank tensorial radiance fields.

Methods

The core of the proposed method lies in understanding and controlling the spectral properties of the radiance field.

  • Spectral Analysis and Gaussian Filtering: The authors conduct a 1D pilot study to demonstrate that naive joint optimization can lead to local optima due to high-frequency signals. They propose using Gaussian filters on both the 2D input images and the 3D radiance field. This filtering acts as a coarse-to-fine strategy, smoothing out high-frequency components initially and allowing for more stable convergence of camera poses and scene geometry.
  • Separable Component-Wise Convolution: To efficiently apply Gaussian filtering to the decomposed low-rank tensor representation (like TensoRF), they introduce a "separable component-wise convolution." This technique leverages the structure of the decomposed tensor, applying convolutions to individual components rather than the entire dense tensor, significantly reducing computational overhead.
  • Robustness Techniques: To further enhance stability, they incorporate:
  • Smoothed 2D Supervision: Blurring input images with scheduled Gaussian kernels.
  • Randomly Scaled Kernel Parameters: Using a range of kernel sizes to adapt to varying scene complexities.
  • Edge-Guided Loss Mask: Amplifying the loss signal on edge regions of images, as these are crucial for pose estimation.

Datasets

The paper evaluates their method on standard datasets for neural rendering:

  • NeRF-Synthetic: A widely used benchmark for novel view synthesis.
  • LLFF (Local Light Field Fusion): Another common dataset for novel view synthesis, often used for evaluating pose estimation robustness.

They also perform experiments on planar image alignment tasks, using cropped image patches from a ground truth image.

Key Insights and Ideas

  • Spectral Bias is Crucial for Robust Joint Optimization: Unlike MLPs, voxel-based representations (even decomposed ones) lack inherent spectral biases. This makes them susceptible to noise and high-frequency details, hindering robust joint pose optimization.
  • Gaussian Filtering as a Coarse-to-Fine Strategy: Applying Gaussian filters to both input images and the 3D scene representation acts as an effective coarse-to-fine mechanism, guiding the optimization towards better solutions.
  • Decomposed Tensor Structure Enables Efficient Filtering: The low-rank tensor decomposition allows for a computationally efficient filtering approach (separable component-wise convolution) that avoids reconstructing a dense tensor.
  • Robustness Techniques are Essential for Real-World Scenarios: While spectral filtering is key, techniques like edge-guided loss and random kernel scaling are vital for handling the complexities and noise present in real-world datasets.

Key Results

  • Improved Novel View Synthesis: The proposed method achieves state-of-the-art novel view synthesis quality on both NeRF-Synthetic and LLFF datasets, often outperforming previous methods in PSNR and LPIPS metrics.
  • Reduced Training Time: The method converges significantly faster, requiring only 20-25% of the training iterations compared to previous state-of-the-art methods (e.g., 50k iterations vs. 200k).
  • Accurate Pose Estimation: The method demonstrates superior or competitive pose registration accuracy across various scenes and datasets.
  • Robustness: Ablation studies confirm the effectiveness of each proposed component (Gaussian filtering, random scaling, edge-guided loss) in improving robustness.

Limitations and Future Work

  • Applicability to Other Tensor Decompositions: While the method is primarily demonstrated with TensoRF's VM-decomposition, the authors note its potential applicability to CP-decomposition.
  • Computational Cost of Filtering: Although more efficient than brute-force 3D convolution, the filtering still adds computational overhead compared to a non-filtered approach.
  • Potential for Further Optimization: The paper suggests that the coarse-to-fine schedule for Gaussian kernel width can be further explored for even better performance.

The authors' work opens avenues for more robust and efficient neural rendering pipelines, particularly in scenarios with unknown or inaccurate camera poses.

Summary updated

Abstract

In this paper, we propose an algorithm that allows joint refinement of camera pose and scene geometry represented by decomposed low-rank tensor, using only 2D images as supervision. First, we conduct a pilot study based on a 1D signal and relate our findings to 3D scenarios, where the naive joint pose optimization on voxel-based NeRFs can easily lead to sub-optimal solutions. Moreover, based on the analysis of the frequency spectrum, we propose to apply convolutional Gaussian filters on 2D and 3D radiance fields for a coarse-to-fine training schedule that enables joint camera pose optimization. Leveraging the decomposition property in decomposed low-rank tensor, our method achieves an equivalent effect to brute-force 3D convolution with only incurring little computational overhead. To further improve the robustness and stability of joint optimization, we also propose techniques of smoothed 2D supervision, randomly scaled kernel parameters, and edge-guided loss mask. Extensive quantitative and qualitative evaluations demonstrate that our proposed framework achieves superior performance in novel view synthesis as well as rapid convergence for optimization. The source code is available at https://github.com/Nemo1999/Joint-TensoRF.
BibTeX
@inproceedings{chen2024improving,
  author = {Bo-Yu Chen and Wei-Chen Chiu and Yu-Lun Liu},
  title = {{Improving Robustness for Joint Optimization of Camera Pose and Decomposed Low-Rank Tensorial Radiance Fields}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=4aj2ecbkMD},
}
Figure