Bidirectional Optical Flow NeRF: High Accuracy and High Quality under Fewer Views

Shuo Chen, Binbin Yan, Xinzhu Sang, Duo Chen, Peng Wang, Xiao Guo, Chongli Zhong, Huaming WanAAAI 2023 · DOI

TL;DR

This paper introduces BOF-NeRF, a novel approach to synthesizing high-quality novel views from sparse input images by leveraging bidirectional optical flow to guide NeRF's scene geometry estimation. It also proposes a view-enhanced fusion method to recover fine details lost in traditional NeRF outputs, significantly improving accuracy and visual fidelity.

What Problem Does This Paper Address?

Synthesizing realistic novel views of a 3D scene from a limited number of input images is a long-standing challenge in computer vision. Traditional Neural Radiance Fields (NeRF) methods, while excellent at view synthesis with abundant data, struggle when presented with sparse views. This scarcity leads to ambiguous depth estimation and consequently, inaccurate scene geometry, resulting in blurry or distorted novel views. Existing solutions often rely on external depth information or depth estimation networks, but these can be unreliable with sparse inputs due to a lack of spatial consistency or poor depth estimation performance. This paper aims to address the core problem of poor scene geometry estimation and novel view detail loss in NeRF when only a few input views are available.

Methods

The core innovation of BOF-NeRF lies in its use of bidirectional optical flow to constrain NeRF's learning process.

  • Bidirectional Optical Flow Neural Radiance Fields: Instead of relying solely on photometric consistency, the method computes forward and backward optical flow between input images. This optical flow information is then used to construct a novel loss function.
  • Optical Flow Estimation: The paper utilizes the RAFT algorithm for robust optical flow estimation between adjacent images.
  • Optical Flow Masking: To handle occlusions and lighting variations, an optical flow mask is generated. This mask identifies valid optical flow areas by considering camera intrinsics and color consistency, ensuring that only reliable flow information is used for supervision.
  • Optical Flow Loss: The computed optical flow is used to supervise the depth estimation within the NeRF framework. By mapping predicted 3D coordinates to adjacent input views and comparing them with ground truth, an optical flow loss is derived. This loss, combined with the standard photometric loss, guides NeRF to learn accurate scene geometry even with sparse views.
  • View-Enhanced Fusion: To combat the loss of fine details in synthesized novel views, a post-processing step is introduced.
  • Coordinate Mapping: The depth image of the synthesized novel view is used to project its pixels back into the original input views (left and right).
  • Detail Refinement: By comparing the color values of these projected pixels with the corresponding pixels in the input views, a fusion weight is calculated. This weight determines how much information from the input views is blended into the novel view, effectively recovering lost details and enhancing sharpness.

Datasets

The paper evaluates its method on three standard datasets:

  • Middlebury 2005 Stereo datasets: Used to test performance under varying baseline distances, specifically on the "Dwarves" scene.
  • NeRF-LLFF dataset: A collection of randomly shot sparse views, used to evaluate synthesis quality with two, four, and six viewpoints on the "Horns" scene.
  • DTU MVS dataset: A large dataset with 128 scenes, used for evaluating performance on the "scan1" scene with two viewpoints.

Key Insights and Ideas

  • Optical Flow as a Geometric Prior: The paper demonstrates that 2D optical flow, when properly masked and formulated as a loss, can act as a powerful geometric prior for implicit neural representations like NeRF, especially in low-data regimes. This is a significant departure from relying solely on photometric consistency.
  • Bridging the Gap in Sparse View Synthesis: By directly addressing the geometry estimation problem with optical flow, BOF-NeRF offers a more robust solution for sparse view synthesis than methods that solely rely on depth supervision from potentially unreliable sources.
  • Detail Recovery through Fusion: The view-enhanced fusion method highlights that even with improved geometry estimation, recovering fine-grained details requires explicit integration of information from neighboring views, a step often overlooked or handled less effectively by prior methods.

Key Results

  • Quantitative Improvements: BOF-NeRF consistently outperforms state-of-the-art baselines (NeRF, pixel-NeRF, DS-NeRF) across various metrics (PSNR, SSIM, LPIPS) on the tested datasets, particularly in sparse view scenarios. For instance, on the Horns dataset with 2-view input, BOF-NeRF achieves a PSNR of 24.75, significantly higher than NeRF's 16.39.
  • Geometric Accuracy: The method shows superior projection error (Perror) on the Middlebury dataset, indicating more accurate scene geometry estimation, especially at larger baseline distances.
  • Qualitative Improvements: Visual comparisons show that BOF-NeRF synthesizes sharper, more detailed novel views with more accurate geometric structures compared to other methods. The view-enhanced fusion is particularly effective in recovering fine textures and edges.

Limitations and Future Work

  • Untextured/Repeatedly Textured Scenes: The authors acknowledge that performance degrades in scenes with uniform or repetitive textures, where optical flow estimation can be less reliable.
  • Interpolated vs. Extrapolated Views: The method excels at synthesizing views that are interpolated between the input views but cannot address extrapolated view synthesis (views outside the input camera's field of view).
  • Future Directions: The authors suggest focusing on addressing extrapolated view synthesis and improving performance in scenes with weak textures.

Summary updated

BibTeX
@inproceedings{chen2023bidirectional,
  author = {Shuo Chen and Binbin Yan and Xinzhu Sang and Duo Chen and Peng Wang and Xiao Guo and Chongli Zhong and Huaming Wan},
  title = {{Bidirectional Optical Flow NeRF: High Accuracy and High Quality under Fewer Views}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i1.25109},
}
Figure