LensNeRF: Rethinking Volume Rendering based on Thin-Lens Camera Model

Min-Jung Kim, Gyojung Gu, Jaegul ChooWACV 2024 · OpenReview

Abstract

Recent advances in Neural Radiance Field (NeRF) show promising results in rendering realistic novel view images. However, NeRF and its variants assume that input images are captured using a pinhole camera and that subjects in images are always all-in-focus by tacit agreement. In this paper, we propose aperture-aware NeRF optimization and rendering methods using a thin-lens model (dubbed LensNeRF), which allows defocus images of any aperture size as input and output. To generalize a pinhole camera model to a thin-lens camera model in NeRF framework, we define multiple rays originating from the aperture area, solving world-to-pixel scale ambiguity. Also, we propose in-focus loss that assigns the given pixel color to points on the focus plane to alleviate the color ambiguity caused by the use of multiple rays. For the rigorous evaluation of the proposed method, we collect a real forward-facing dataset with different F-numbers for each viewpoint. Experimental results demonstrate that our method successfully fuses an aperture-size adjustable thin-lens camera model into the NeRF architecture, showing favorable qualitative and quantitative results compared to baseline models.

Introduction

Standard Neural Radiance Fields (NeRF) have revolutionized novel view synthesis but rely heavily on the pinhole camera model. This model assumes that every point in a scene is perfectly in focus, regardless of its distance from the camera. In real-world photography, however, cameras use lenses with finite apertures that create a "depth-of-field" effect. This results in defocus blur for objects outside the focus plane.

When standard NeRFs encounter training images with shallow depth-of-field, they often fail by attempting to represent blur as "cloudy" or "foggy" geometry. Furthermore, they lack the mathematical framework to synthesize realistic "bokeh" or adjust the aperture (F-number) after training. LensNeRF addresses these limitations by integrating a physical thin-lens model directly into the volume rendering pipeline.

Methodology

LensNeRF bridges the gap between computer vision and physical optics through several key technical innovations:

Thin-Lens Ray Modeling

Instead of casting a single ray per pixel as in a pinhole model, LensNeRF samples multiple rays originating from different points across a simulated aperture area. These rays are refracted based on the thin-lens equation and converge at the focus plane. This allows the model to physically simulate how light travels through a lens.

The LensNeRF architecture integrates a thin-lens model by sampling multiple rays per pixel and aggregating them through a voxel-based volume rendering pipeline.

Scale Calibration and In-Focus Loss

A significant challenge in simulating lenses is scale ambiguity. NeRFs typically operate in arbitrary coordinate systems, but a physical lens model requires knowing the relationship between pixels and world units (e.g., meters). LensNeRF uses sparse 3D points from COLMAP to estimate in-focus depth and optimizes a scale factor ($s_{3D \rightarrow pix}$) to link these units accurately.

To further improve reconstruction, the authors introduce an In-Focus Loss. Since a blurred pixel is a mixture of many colors, determining the "true" color is an ill-posed problem. This loss function gives higher weight to the color consistency of points located on the focus plane, using the sharpest parts of the image to guide the reconstruction of the entire scene.

Voxel-Based Backbone

The method is built on Direct Voxel Grid Optimization (DVGO). By using explicit voxel grids for density and features, LensNeRF maintains the efficiency required to handle the increased computational load of sampling multiple rays per pixel.

Experimental Results

To evaluate the model, the researchers introduced a new real-world forward-facing dataset captured with a Canon EOS 550D, featuring nine scenes with varying aperture settings (F4 to F22). LensNeRF was tested on two primary tasks:

  1. Defocus Task: Converting sharp (narrow-aperture) inputs into blurry (wide-aperture) outputs. LensNeRF significantly outperformed baselines like NeRFocus and DoF-NeRF in visual quality.
  2. Deblur Task: Recovering a sharp scene from blurry inputs. LensNeRF achieved a PSNR of 26.46 on the F4 to F22 task, compared to 23.44 for Deblur-NeRF.
LensNeRF enables the synthesis of novel views with adjustable aperture sizes, allowing for both deblurring (narrow aperture) and realistic bokeh effects (wide aperture).

Conclusion

LensNeRF successfully demonstrates that integrating physical camera models into neural rendering pipelines leads to more robust and interpretable scene representations. By moving beyond the pinhole assumption, the model can handle real-world photographic effects like defocus blur and provide users with post-capture control over aperture and focus. While the multi-ray sampling increases computational cost and requires EXIF metadata, the gains in image quality and physical accuracy mark a significant step forward for NeRF research.

Summary updated

Abstract

Recent advances in Neural Radiance Field (NeRF) show promising results in rendering realistic novel view images. However, NeRF and its variants assume that input images are captured using a pinhole camera and that subjects in images are always all-in-focus by tacit agreement. In this paper, we propose aperture-aware NeRF optimization and rendering methods using a thin-lens model (dubbed LensNeRF), which allows defocus images of any aperture size as input and output. To generalize a pinhole camera model to a thin-lens camera model in NeRF framework, we define multiple rays originating from the aperture area, solving world-to-pixel scale ambiguity. Also, we propose in-focus loss that assigns the given pixel color to points on the focus plane to alleviate the color ambiguity caused by the use of multiple rays. For the rigorous evaluation of the proposed method, we collect a real forward-facing dataset with different F-numbers for each viewpoint. Experimental results demonstrate that our method successfully fuses an aperture-size adjustable thin-lens camera model into the NeRF architecture, showing favorable qualitative and quantitative results compared to baseline models. The dataset will be made available.
BibTeX
@inproceedings{kim2024lensnerf,
  author = {Min-Jung Kim and Gyojung Gu and Jaegul Choo},
  title = {{LensNeRF: Rethinking Volume Rendering based on Thin-Lens Camera Model}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=MMXSQQvDfc},
}
Figure