Camera-Independent Single Image Depth Estimation From Defocus Blur

Lahiru Wijayasingha, Homa Alemzadeh, John A. StankovicWACV 2024

TL;DR

This paper tackles the challenge of camera dependency in single-image depth estimation from defocus blur. They introduce a novel method that leverages optical physics to create camera-independent depth estimation models, significantly improving robustness across different cameras without retraining.

What Problem Does This Paper Address?

Estimating depth from a single image is a crucial task in computer vision, with applications ranging from augmented reality to robotics. One promising approach is using defocus blur, where the degree of blur in an image can indicate an object's distance. However, existing methods that rely on defocus blur are highly sensitive to the specific camera used to capture the images. This means a model trained on images from one camera will perform poorly when tested on images from another, even if the scene is identical. This camera dependency arises because different cameras, with varying lens apertures, focal lengths, and sensor characteristics, produce defocus blurs with different properties. The paper aims to overcome this limitation by developing a camera-independent depth estimation technique.

Methods

The core of the paper's contribution lies in its approach to making depth estimation from defocus blur camera-independent.

  • Optical Physics Modeling: The authors start by analyzing the optical physics equations that govern defocus blur. They demonstrate how parameters like f-number, focal length, and pixel size influence the blur's characteristics.
  • Camera Parameter Estimation (Defocus Blur Calibration): A key innovation is a novel "Defocus Blur Calibration" method. This technique allows for the estimation of camera-specific parameters, particularly a parameter they call $k_{cam}$, which encapsulates the camera's influence on defocus blur. This calibration process involves capturing pairs of focused and blurred images of a calibration pattern and analyzing the blur characteristics of the pattern's edges.
  • Camera-Independent Depth Estimation Model: The proposed depth estimation model is designed to be camera-independent. It consists of two main CNN-based components:
  • Blur Estimation: This module estimates the standard deviation of the Point Spread Function (PSF) at each pixel, representing the total blur.
  • Depth Estimation: This module takes the estimated blur information and, crucially, the camera-specific parameter $k_{cam}$ as input. By dividing the estimated blur by $k_{cam}$, the model effectively normalizes the blur, making it independent of the specific camera. This normalized blur is then used to predict depth. The authors also incorporate skip connections to inject image features into the depth estimation process, which helps handle ambiguities in depth prediction.
  • Loss Function: The training process uses a combined loss function that includes both blur estimation loss and depth estimation loss, with a weighting parameter ($b_{weight}$) to balance their contributions.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Defocusnet Dataset: A synthetic dataset generated by Maximov et al. [19], containing focal stacks with varying $k_{cam}$ values.
  • Synthetic Blender Dataset: A new synthetic dataset created by the authors, expanding on the Defocusnet dataset with more realistic textures and various simulated cameras.
  • DDFF12 Dataset: A real-world dataset from Hazirbas et al. [11], consisting of images captured with a lightfield camera.
  • NYU Depth v2 Dataset: A widely used real-world dataset containing aligned RGB and depth image pairs. The authors artificially create defocus blurred images from this dataset.

Key Insights and Ideas

  • Defocus Blur is Camera-Dependent: The fundamental insight is that the same physical blur amount can be produced by different camera settings, and conversely, the same camera setting can produce different blur amounts depending on the scene. This dependency is quantified by the $k_{cam}$ parameter.
  • Optical Physics as a Bridge: Leveraging optical physics equations provides a principled way to understand and model the camera's influence on defocus blur, moving beyond purely data-driven approaches.
  • Calibration is Key to Generalization: The "Defocus Blur Calibration" method is crucial. By estimating $k_{cam}$, the model can adapt to new, unseen cameras without requiring retraining, demonstrating a significant step towards generalization.
  • Defocus Blur is More Robust than Semantic Methods: The paper argues that defocus blur-based depth estimation is more robust to domain changes compared to semantic-based methods, as blur is a local image feature.

Key Results

  • Improved Accuracy: The proposed camera-independent method achieves significantly lower errors compared to state-of-the-art methods on various datasets. For instance, they report error reductions of around 3cm on DDFF12, 7cm on NYU Depth v2, and 5cm on their synthetic dataset.
  • Robustness to Camera Changes: Experiments show that their method, when using estimated $k_{cam}$ values, performs much better than models that ignore camera parameters (No $k_{cam}$) when tested on images from different cameras.
  • Effective Calibration: The Defocus Blur Calibration method demonstrates accurate estimation of $k_{cam}$ values, with estimated values closely matching ground truth values in experiments.
  • Performance Gains: On the NYU Depth v2 dataset, their method (GT $k_{cam}$ and est $k_{cam}$) outperforms the VPD model by around 0.04 in RMSE, translating to a 4cm error reduction.

Limitations and Future Work

  • Field of View (FOV) Sensitivity: The authors acknowledge that their current model does not explicitly account for changes in the Field of View (FOV) that can occur with different camera settings (e.g., focal length changes). This limitation leads to a significant performance drop when images are resized, as the image features are not adapted. Future work could address this by modifying image features to reflect FOV changes.
  • Close-Range Effectiveness: The paper notes that the sensitivity of blur to distance diminishes at larger ranges, limiting the effectiveness of defocus blur techniques to close-range measurements (typically under 2-3 meters).
  • Privacy and Security Concerns: The authors raise ethical considerations, noting that techniques like theirs could potentially be misused for surveillance or harmful applications. They emphasize the importance of addressing these concerns alongside algorithmic advancements.

Summary updated

BibTeX
@inproceedings{wijayasingha2024camera,
  author = {Lahiru Wijayasingha and Homa Alemzadeh and John A. Stankovic},
  title = {{Camera-Independent Single Image Depth Estimation From Defocus Blur}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure