Selective Light Field Refocusing for Camera Arrays Using Bokeh Rendering and Superresolution

Yingqian Wang, Jungang Yang, Yulan Guo, Chao Xiao, Wei AnIEEE Signal Processing Letters 2019 · DOI · OpenReview

TL;DR

This paper introduces a novel method for refocusing light field images captured by camera arrays. It simultaneously enhances the sharpness of focused regions through super-resolution and artistically renders the out-of-focus areas (bokeh), offering adjustable depth-of-field control post-capture. This approach aims to bridge the gap between the high-quality imaging capabilities of professional DSLRs and the accessibility of modern camera arrays.

What Problem Does This Paper Address?

Traditional DSLRs excel at producing aesthetically pleasing images with shallow depth of field (DoF), where specific subjects are in sharp focus while the background is blurred into a desirable bokeh. This effect guides viewer attention and enhances image aesthetics. However, professional DSLRs are often bulky and expensive. While light field (LF) imaging, particularly using camera arrays, offers a path to capture spatial and angular information with potentially higher resolution than plenoptic cameras, existing refocusing methods for camera arrays struggle with aliasing artifacts in bokeh regions and often cannot simultaneously improve resolution and render pleasing bokeh. Furthermore, the ability to adjust the DoF after the image has been captured is a desirable feature for creative control, which is typically not available with standard camera setups.

Methods

The proposed method integrates two key processes: bokeh rendering and super-resolution (SR).

  1. Disparity Estimation: The process begins by estimating a disparity map, which is crucial for understanding the depth information within the scene.
  2. Bokeh Rendering: An anisotropic filter is employed to render the out-of-focus regions. This filter uses depth information to map source pixels to circles of confusion (CoC) and blends them based on depth order. The radius of the CoC is calculated based on the focal length, F-number, baseline, and disparity, allowing for control over the bokeh intensity (represented by a parameter 'K').
  3. Reconstruction-Based Super-Resolution: A reconstruction-based SR approach is used to enhance the resolution of the focused regions. This model accounts for degradation factors like blurring, down-sampling, and shifts inherent in camera array captures. Crucially, the rendered bokeh image is incorporated as a regularization term within the SR optimization problem. This ensures that the SR process not only sharpens the in-focus areas but also maintains the aesthetic quality of the bokeh.

The method allows for post-adjustment of the depth of field by simply changing the bokeh intensity parameter 'K' during the rendering phase, offering flexibility in image editing.

Datasets

The authors evaluated their method on:

  • Public Datasets: The Stanford LF dataset [24] was used, specifically the 3x3 sub-images which were down-sampled by a factor of 2 for testing.
  • Self-Developed Dataset: A custom dataset was created using an iPhone 6S camera mounted on a gantry, capturing scenes in a laboratory environment. This dataset was calibrated, and "groundtruth" images were captured using a Canon EOS 5D Mark IV DSLR. The HCI LF dataset [27] was also used to evaluate the influence of disparity estimation errors.

Key Insights and Ideas

  • Synergistic Integration: The core innovation lies in the selective integration of bokeh rendering and super-resolution into a single scheme. This allows for simultaneous improvement of image resolution in focused areas and aesthetic rendering of blurred regions, addressing limitations of prior methods that focused on one aspect or the other.
  • Post-Capture DoF Adjustment: The ability to adjust the depth of field after an image has been captured is a significant advantage, offering creative control that is not possible with DSLRs or most previous LF refocusing techniques.
  • Bokeh as Regularization: Using the rendered bokeh image as a regularization term in the SR process is a clever way to ensure consistency between the focused and blurred regions, leading to more natural-looking refocused images.
  • Anisotropic Filtering for Bokeh: The use of an anisotropic filter for bokeh rendering, considering CoC and depth, contributes to aesthetically pleasing blur effects.

Key Results

  • Visual Performance: The proposed method achieved superior visual performance compared to existing methods [9], [14], [20] on both public and self-developed datasets. It produced refocused images with higher PSNRs in focused regions and more aesthetically pleasing bokeh.
  • Bokeh Quality: The bokeh rendered by the proposed method was found to be similar to that generated by a DSLR, a significant achievement for low-cost camera arrays.
  • Adjustable DoF: Experiments demonstrated the ability to control the depth of field by adjusting the 'K' parameter, allowing for shallow or deep DoF effects as needed.
  • Computational Cost: The method offers an acceptable computational cost, being significantly faster than some state-of-the-art LF reconstruction methods [14] (reportedly 5-10% of the execution time).
  • Disparity Error Robustness: While disparity estimation errors can introduce minor artifacts, the method shows robustness to these inaccuracies, particularly on boundaries.

Limitations and Future Work

  • Disparity Estimation Influence: The paper acknowledges that the visual performance of their method is partially influenced by the accuracy of the estimated disparity. Errors in disparity estimation, especially on boundaries, can lead to artifacts.
  • Computational Cost vs. Resolution: While faster than some methods, the execution time still increases with image resolution and the number of iterations (NoI). The paper notes that NoI has a greater influence on execution time than the bokeh parameter 'K'.
  • Natural Gap: The authors admit that while their method significantly improves image quality, the focused regions might not be as sharp as those from a high-end DSLR, acknowledging the inherent limitations of low-cost camera arrays.

Future work could focus on further improving the accuracy and robustness of the disparity estimation step, potentially exploring end-to-end learned approaches for combined SR and bokeh rendering, and investigating methods to further reduce computational complexity for real-time applications.

Summary updated

Abstract

Camera arrays provide spatial and angular information within a single snapshot. With refocusing methods, focal planes can be altered after exposure. In this letter, we propose a light field refocusing method to improve the imaging quality of camera arrays. In our method, the disparity is first estimated. Then, the unfocused region (bokeh) is rendered by using a depth-based anisotropic filter. Finally, the refocused image is produced by a reconstruction-based superresolution approach where the bokeh image is used as a regularization term. Our method can selectively refocus images with focused region being superresolved and bokeh being aesthetically rendered. Our method also enables postadjustment of depth of field. We conduct experiments on both public and self-developed datasets. Our method achieves superior visual performance with acceptable computational cost as compared to other state-of-the-art methods. Code is available at https://github.com/YingqianWang/Selective-LF-Refocusing.
BibTeX
@article{wang2019selective,
  author = {Yingqian Wang and Jungang Yang and Yulan Guo and Chao Xiao and Wei An},
  title = {{Selective Light Field Refocusing for Camera Arrays Using Bokeh Rendering and Superresolution}},
  journal = {IEEE Signal Processing Letters},
  year = {2019},
  doi = {10.1109/lsp.2018.2885213},
}
Figure