Abstract
The problem of reconstructing three-dimensional (3D) scene geometry and radiometry from images is an important problem in computer vision and has applications in a variety of fields such as medicine and artifact preservation. However, state-of-the-art multiview algorithms assume a pinhole camera that incorrectly models defocus blur as a property of the scene instead of a property of the imaging process. We address the problem of dense 3D shape reconstruction from multiple viewpoints in situations where the image data exhibits noticeable defocus. We develop a mathematical framework for a fully generative variational algorithm that iteratively deforms an estimate of the foreground surface shapes and scene radiance such that irradiance estimates given by the thin lens forward model are photometrically consistent with the actual image data. This framework is founded on novel geometric computations of flux differentials across an evolving surface as well as gradients along occluding boundaries and their projections. While more work is needed to make them fit for practical use, the future potential of methods based on these computations is shown with experiments reconstructing simple object shapes from both synthetically generated and real defocused images. While our reconstruction algorithm has a higher computational cost than pinhole-based methods due to the more general optical model, it better reconstructs object proportions as well as sharp features that are blurred due to image defocus. As such, our geometry-based method provides a unified framework that extends the applicability of multiview reconstruction techniques to the poorly supported domain of defocused images where state-of-the-art pinhole-based methods fail.
Introduction
Most state-of-the-art multiview 3D reconstruction algorithms rely on the pinhole camera model. While mathematically convenient, pinholes assume an infinite depth of field where all scene points are perfectly in focus. In the real world, lenses have a limited depth of field; objects too close or too far from the focal plane appear blurred. When a pinhole-based algorithm encounters this "defocus blur," it often treats the blur as a property of the object's surface (like a fuzzy texture) rather than a byproduct of the imaging process. This leads to "thickened" reconstructions, rounded corners, and a general failure to reconcile different levels of blur across multiple views. This paper introduces a framework that bridges the gap between Multiview Stereo and Depth-from-Defocus by replacing the pinhole assumption with a more realistic thin lens model.
Methodology
The authors propose a generative variational framework. Instead of simple pixel matching, the algorithm maintains an evolving 3D estimate of the scene's shape and radiance, then "renders" what this scene should look like through a thin lens to compare it against captured images.
Thin Lens Forward Model
Unlike a pinhole that projects a single ray per pixel, the thin lens model integrates a cone of light rays passing through the lens aperture. As shown in Figure 1, a point $P(X)$ on the focal plane projects through the lens $O$ of diameter $d$. For points off the focal plane, the resulting circle of confusion is explicitly modeled.
Variational Evolution
The algorithm uses gradient descent to deform a 3D surface, represented via level sets, until the predicted images match the real data. The core mathematical contribution is the derivation of "flux differentials," which allow the algorithm to calculate how image irradiance changes as the 3D surface moves, specifically accounting for complex areas like occluding boundaries in the presence of blur.
Key Results and Discussion
The framework was tested on synthetic datasets (tetrahedron and dumbbell) and real-world images.
Accuracy and Robustness
In the tetrahedron test, the pinhole model failed to reconcile different blur levels, creating a "conglomeration" of shapes. In contrast, the thin lens method reconstructed a single, coherent 3D object. Furthermore, the thin lens model provides "natural" smoothing; because it averages light over a cone of rays, image noise is naturally filtered out during reconstruction, whereas pinhole models often require artificial mathematical penalties to maintain surface smoothness.
Computational Trade-offs
The primary limitation is computational cost. Calculating light integration for every pixel across an evolving surface is expensive, with the thin lens method taking significantly longer per iteration than pinhole-based approaches. However, the thin lens method often requires fewer iterations to reach a stable, accurate shape and achieves significantly lower total squared error.
Conclusion
By uniting stereo and depth-from-defocus cues within a thin lens variational framework, this method extends the applicability of multiview reconstruction to defocused imagery. While currently limited to piecewise constant radiance and higher computational loads, it demonstrates the potential for high-fidelity 3D reconstruction in challenging optical conditions where standard pinhole-based methods fail.
Summary updated