Variational Depth from Focus Reconstruction

Michael Möller, Martin Benning, Carola-Bibiane Schönlieb, Daniel CremersIEEE TIP 2014 · DOI · OpenReview

TL;DR

This paper introduces a novel variational approach for reconstructing depth maps from a sequence of differently focused images (Depth from Focus - DFF). By formulating the problem as a non-convex energy minimization with a convex regularization, and solving it efficiently using a linearized ADMM, the method achieves more robust and realistic depth maps compared to traditional techniques, especially in challenging scenarios with noise and textureless regions.

What Problem Does This Paper Address?

Reconstructing a 3D scene's depth map from a series of images taken at different focal settings (Depth from Focus or Shape from Focus) is a challenging computer vision problem. The core idea is that a pixel is sharpest at a specific focal distance corresponding to its depth. However, this estimation is inherently ill-posed, especially in areas with little texture, leading to noisy and inaccurate depth maps. Existing methods often rely on maximizing contrast measures, which can be sensitive to noise and may struggle with complex scene structures. This paper aims to address these limitations by proposing a more robust and principled variational framework that can handle noise and produce smoother, more accurate depth estimates.

Methods

The authors propose a variational approach that frames depth map reconstruction as an energy minimization problem. The energy function consists of two main components:

  • Data Fidelity Term (Non-convex): This term captures the relationship between the image data and the estimated depth. It's designed to be smooth but non-convex, making it robust to noise but computationally challenging to minimize.
  • Regularization Term (Convex and Non-smooth): This term enforces desired properties on the depth map, such as smoothness. The paper uses the discrete isotropic Total Variation (TV) regularization, which encourages piecewise constant depth maps and is known for its ability to preserve edges.

To efficiently solve this non-convex minimization problem, the authors employ a linearized Alternating Directions Method of Multipliers (ADMM). This technique introduces an auxiliary variable and iteratively updates the depth map and the auxiliary variable, along with a dual variable, to converge to a solution. The linearization of the non-convex data fidelity term is key to making the subproblems solvable efficiently.

Datasets

The paper evaluates its method on two types of datasets:

  • Simulated Data: Generated using a publicly available Matlab code (from [9] and [4]) that simulates focus blur and adds Gaussian noise to an input image. This allows for quantitative evaluation against ground truth depth maps.
  • Real Data:
    • A dataset from Said Pertuz' website, consisting of 33 color images with 640x480 resolution.
  • A high-quality dataset from an ARRI Alexa camera, comprising 373 images of a tabletop scene, downscaled to 1080x1920.

Key Insights and Ideas

  • Variational Framework for DFF: Framing DFF as a variational energy minimization problem allows for a more principled way to incorporate prior knowledge (regularization) and handle the ill-posed nature of the problem.
  • Non-convex Data Fidelity + Convex Regularization: This combination is crucial. The non-convexity of the data term helps in robustness to noise, while the convex TV regularization ensures smooth and structured depth maps.
  • Linearized ADMM for Efficiency: Directly minimizing the non-convex energy is difficult. Linearizing the data fidelity term within an ADMM framework makes the optimization tractable and efficient, enabling practical application.
  • Robustness to Textureless Regions: The proposed method handles textureless areas better than classical approaches by not relying solely on sharp contrast peaks, which are absent in such regions. The variational approach leads to a "fuzzy" decision for pixel reliability, avoiding abrupt errors.
  • Joint Denoising and Reconstruction: The variational approach inherently performs denoising as part of the depth reconstruction process, unlike methods that first estimate depth and then apply a separate denoising step.

Key Results

  • Superior Performance on Simulated Data: The variational approach consistently outperformed classical MLAP-based methods (with varying window sizes and median filtering) across different simulated depth map shapes (cone, plane, cosine, sphere), achieving lower Root Mean Square Error (RMSE) values.
  • More Realistic Depth Maps on Real Data: Qualitative results on real-world datasets show that the variational approach produces smoother and more visually plausible depth maps, especially in challenging areas where classical methods still exhibit significant errors.
  • Efficient Computation: The linearized ADMM algorithm converges quickly, with the total computational time for depth estimation and minimization being less than three seconds for a sequence of 33 images (640x480) on a GPU. The actual minimization step takes less than 0.2 seconds.
  • Convergence: The algorithm demonstrates monotonic decay of the total energy and successive iterates, indicating convergence to a critical point.

Limitations and Future Work

  • Convexity Assumption for Convergence Proof: While the paper provides a convergence result for the convex case, the theoretical guarantees for the non-convex scenario are still an open research problem. The authors note that their method numerically results in a stable optimization scheme.
  • Loss of Contrast: The paper acknowledges that Total Variation regularization can sometimes lead to a loss of contrast in the reconstructed depth maps.
  • Future Work: The authors plan to incorporate more sophisticated regularizations and explore correcting the contrast loss by applying nonlinear Bregman iterations. They also mention the possibility of extending their work to handle more complex scenarios and potentially real-time applications.

Summary updated

Abstract

This paper deals with the problem of reconstructing a depth map from a sequence of differently focused images, also known as depth from focus or shape from focus. We propose to state the depth from focus problem as a variational problem including a smooth but nonconvex data fidelity term, and a convex nonsmooth regularization, which makes the method robust to noise and leads to more realistic depth maps. Additionally, we propose to solve the nonconvex minimization problem with a linearized alternating directions method of multipliers (ADMM), allowing to minimize the energy very efficiently. A numerical comparison to classical methods on simulated as well as on real data is presented.
BibTeX
@article{moller2014variational,
  author = {Michael Möller and Martin Benning and Carola-Bibiane Schönlieb and Daniel Cremers},
  title = {{Variational Depth from Focus Reconstruction}},
  journal = {IEEE Trans. Image Process.},
  year = {2014},
  doi = {10.1109/tip.2015.2479469},
}
Figure