Abstract
DeepLens is a neural network-based system designed to generate high-quality, adjustable shallow depth-of-field (DoF) effects from a single "all-in-focus" image. By integrating a multi-task depth prediction module, a memory-efficient lens blur module, and a recurrent guided upsampling network, DeepLens enables interactive focal point and aperture adjustments with significantly fewer artifacts than existing single-image or dual-lens methods.
Introduction
The "bokeh" effect—characterized by a sharp subject against an artistically blurred background—is a hallmark of professional photography typically requiring expensive DSLR cameras with large-aperture lenses. While modern smartphones simulate this via "Portrait Modes" using dual cameras or specialized sensors, these hardware-dependent solutions often suffer from "depth leakage" (blurring subject edges) or "halos" around foreground objects. Furthermore, they offer limited post-capture flexibility. DeepLens addresses these challenges by synthesizing complex optical effects from a single standard image, necessitating accurate depth estimation from 2D data and spatially-varying blur rendering without visual glitches.
Methodology
The DeepLens pipeline consists of three differentiable stages designed for end-to-end training:
- Depth Prediction Module: Utilizing a ResNet-50 backbone, this module predicts a normalized disparity map. To ensure sharp object boundaries—crucial for realistic blur—the authors employ multi-task learning, training the network to simultaneously perform foreground segmentation. This acts as a regularizer, improving depth accuracy at object edges.
- Lens Blur Module: To avoid the high computational and memory costs of standard image-space convolution with large kernels, DeepLens operates in a learned feature space. The network predicts $1 \times 1$ kernels applied to a compressed feature map, which efficiently captures complex blur characteristics with a fraction of the memory required for traditional methods.
- Guided Upsampling Module: To maintain a low memory footprint while producing high-resolution results, the system first generates a low-resolution blurred image. This is then recurrently upsampled using the original high-resolution "all-in-focus" image as a guide to preserve sharp details in focused regions and smooth gradients in the bokeh.
Additionally, a specialized loss function ($J_f$) prioritizes "hard examples" at depth discontinuities, forcing the model to produce cleaner transitions between focused and defocused areas.
Datasets and Evaluation
Training utilized a combination of real-world and synthetic data to optimize different system components:
- iPhone Depth Dataset: 2,462 RGB-D pairs captured with an iPhone dual-lens camera, used primarily to train the depth prediction module on diverse real-world scenes.
- Synthetic Shallow DoF Dataset: 18,000 samples created by compositing foreground objects onto background images with precise artificial depth planes. This allowed the rendering components to learn the physics of blur without the noise inherent in real-world depth sensors.
Key Results
DeepLens achieved a PSNR of 28.235 and an SSIM of 0.908 on the iPhone test set, significantly outperforming direct regression baselines and other state-of-the-art depth-plus-rendering combinations. Qualitative user studies indicated a strong preference for DeepLens over the iPhone's native dual-camera mode, particularly regarding complex boundaries like hair or thin structures. The system is efficient enough for interactive use, processing a 2048px resolution image in approximately 0.7 seconds on a modern GPU.
Conclusion
DeepLens demonstrates that blurring in a compressed feature space and using segmentation as a regularizer for depth estimation significantly improves synthetic DoF quality. While challenges remain with extremely thin structures (e.g., fine branches) where depth estimation may fail, the system provides a robust framework for single-image depth-of-field manipulation. Future work may explore higher-resolution end-to-end training and more complex lens models beyond the standard thin-lens approximation.
Summary updated