Abstract
Inserting a patterned occluder at the aperture of a camera lens has been shown to improve the recovery of depth map and all-focus image compared to a fully open aperture. However, design of the aperture pattern plays a very critical role. Previous approaches for designing aperture codes make simple assumptions on image distributions to obtain metrics for evaluating aperture codes. However, real images may not follow those assumptions and hence the designed code may not be optimal for them. To address this drawback we propose a data driven approach for learning the optimal aperture pattern to recover depth map from a single coded image. We propose a two stage architecture where, in the first stage we simulate coded aperture images from a training dataset of all-focus images and depth maps and in the second stage we recover the depth map using a deep neural network. We demonstrate that our learned aperture code performs better than previously designed codes even on code design metrics proposed by previous approaches.
Introduction
Recovering depth from a single image is a fundamental challenge in computer vision. Depth from Defocus (DfD) techniques leverage the relationship between distance and blur, but standard circular apertures often produce uniform blur that makes depth levels difficult to distinguish. Coded apertures—masks with specific patterns—can encode more discriminative depth information. Traditionally, these patterns were designed using mathematical models based on natural image statistics, such as the "1/f law." However, these assumptions often fail in real-world scenarios, particularly in non-textured regions where traditional methods lack sufficient signal to estimate depth accurately.
Proposed Method
The authors propose an end-to-end framework that jointly optimizes the physical aperture pattern and a deep recovery network. The architecture consists of two main stages:
- Code Selection Stage: This stage simulates the hardware. A learnable weight matrix is passed through a sigmoid function to generate a differentiable mask. This mask is used to simulate coded aperture images by blurring all-focus images according to their corresponding depth maps.
- Depth Prediction Network: A Convolutional Neural Network (CNN) processes the simulated coded images. To better capture depth information, which is often encoded in the frequency domain, the network takes Fourier Transform coefficients as input.
By training these stages together, the system co-designs the hardware and software, allowing the aperture pattern to evolve into a form that maximizes the performance of the recovery network.
Experimental Setup
The model was trained and evaluated using the NYU Depth Dataset V2, which provides RGB images and ground-truth depth maps. The dataset was split into training (60%), validation (20%), and testing (20%) sets. Continuous depth values were discretized into specific bins based on a thin-lens model, framing the depth recovery as a classification task.
Results and Discussion
The data-driven approach yielded several significant improvements over traditional methods:
- Metric Performance: The learned code naturally maximized metrics like Kullback-Leibler (KL) distance, achieving values nearly 10 times higher than previous state-of-the-art designs (e.g., Levin’s or Zhou’s codes).
- Robustness to Texture: Unlike traditional methods that rely heavily on high-contrast textures, the CNN-based approach demonstrated superior performance in smooth, low-texture regions.
- Visual Fidelity: The proposed method produces sharper depth boundaries and identifies smaller objects more accurately than standard monocular depth estimation techniques.
Conclusion
This work demonstrates the power of hardware-software co-design in computational photography. By letting a neural network optimize the physical aperture, the system overcomes the limitations of hand-crafted mathematical models. Future work includes testing the learned codes on physical camera systems to account for real-world noise and aberrations, as well as extending the model to continuous depth estimation.
Summary updated