In computer vision, we frequently model cameras as ideal pinholes. In a theoretical pinhole camera, every point in the 3D world maps to a perfectly sharp pixel on the image sensor, regardless of its distance. Real cameras, however, rely on lenses with finite apertures to gather enough light. This introduces a physical limitation known as a limited depth of field: only objects sitting exactly on a specific 3D plane (the focal plane) project as perfectly sharp points.
Objects positioned in front of or behind this focal plane project onto the sensor as blurred regions rather than sharp points. This blurred shape is known as the Circle of Confusion (CoC), and its size is governed by the physics of the thin-lens equation. If the camera's focal length, aperture, and focus distance are known, the radius of this blur is a direct mathematical function of the object's physical distance from the lens.
This is the core intuition behind Depth from Defocus (DfD): optical blur is not an unwanted image degradation to be eliminated, but a rich geometric signal that implicitly encodes the three-dimensional structure of a scene.
Blur as a Geometric Signal
Historically, computer vision algorithms treated defocus blur as noise or an artifact to be corrected via deblurring. The foundational shift toward treating blur as a constructive signal began by recognizing that spatial variations in sharpness map directly to scene geometry. Alex Pentland demonstrated in A New Sense for Depth of Field that the focal gradient provides a mathematically consistent signal for passive depth sensing. By measuring the lens's point spread function (PSF), a system can calculate distances from a single viewpoint, completely bypassing the computationally expensive feature-matching problems inherent to stereo vision.
Even when blur is subtle or nearly imperceptible to human observers, it contains structural information. For instance, McCloskey et al. showed that the spatial variation of defocus across a single image can reveal the 3D orientation (slant and tilt) of a textured surface. Instead of requiring multiple exposures under different camera settings, the local gradients of blur across a surface provide a primary signal for geometric inference.
Differentiable Optics in Deep Learning
With the advent of deep learning, researchers transitioned from hand-crafted frequency filters to embedding physical optical constraints directly into neural network architectures.
A persistent challenge in monocular depth estimation is that standard deep networks often rely on semantic heuristics—such as assuming the sky is far away or that objects lower in the frame are closer. Consequently, these models frequently fail when encountering unfamiliar compositions or domain shifts. Defocus cues, by contrast, are domain-invariant because they depend strictly on the physics of light propagation. As demonstrated in Focus on Defocus, models trained to leverage physical blur characteristics generalize far better from synthetic training data to real-world environments because they track relative differences in sharpness rather than memorizing scene layouts.
To train these networks without relying on expensive ground-truth depth data (such as LiDAR), frameworks utilize unsupervised reconstruction losses grounded in lens physics. Gur et al. introduced a differentiable Point Spread Function (PSF) convolutional layer into a neural network. By predicting a depth map and using it to mathematically "re-render" a sharp image back into its observed blurry state, the network learns to infer depth purely by minimizing the discrepancy between the synthesized and observed blur.
[Input Blurry Image] ---> [Neural Network] ---> [Predicted Depth Map]
|
v
[Synthesized Sharp Image] <--- [Differentiable PSF Layer]
|
v
[Reconstructed Blurry Image] <--- (Compare via Loss) ---> [Input Blurry Image]
Similarly, Alzayer mapped metric depth into explicit Circle of Confusion maps using optical formulas to guide post-capture refocusing, treating spatially varying blur as a strict mathematical function of depth and lens parameters.
Resolving Ambiguities with Specialized Hardware
While a single blurry image contains rich geometric data, it presents an inherent physical ambiguity: an object produces the exact same blur radius whether it is positioned a certain distance in front of or behind the focal plane. This is known as the sign ambiguity. Furthermore, determining the absolute physical scale of a scene from a single uncalibrated image is mathematically ill-posed. Modern systems resolve these ambiguities by leveraging specialized hardware configurations, such as focal stacks and dual-pixel sensors.
Focal Stacks
A focal stack consists of multiple images captured from the same viewpoint but with different focus settings. By analyzing how the sharpness of an object changes across different slices of the stack, the sign ambiguity is eliminated. The authors of Focal Split implemented a bio-inspired snapshot design that splits incoming light onto differential focal planes simultaneously. This setup allows depth to be calculated via a physically grounded formula that relates image intensity derivatives directly to distance, bypassing traditional gradient-based blur estimation.
Dual-Pixel Sensors
Dual-Pixel (DP) hardware, found in most modern smartphone sensors, splits each pixel into two independent sub-photiodiode halves (left and right). This configuration effectively captures two sub-aperture views of the scene within a single exposure.
As explored in StereoDPNet, the phase discrepancy (or parallax) between these sub-pixels is concentrated precisely in out-of-focus regions, turning defocus blur into a micro-stereo depth cue. In regions where traditional sharp-edge matching fails due to a lack of high-contrast texture, the variation in blur between the sub-pixel views provides a continuous signal for depth inference.
Ashida et al. demonstrated that mathematically coupling these dual-pixel defocus cues with multi-view geometry completely resolves the absolute scale ambiguity of 3D reconstructions without requiring external reference objects or pre-calibrated camera positions. Additionally, Chen et al. showed that the relationship between relative blur kernel diameters and stereo disparity can be modeled as a second-order polynomial, allowing asymmetric stereo pairs to calculate depth without explicit camera calibration for focal length or aperture.
Grounding 3D Radiance Fields in Physical Blur
The latest evolution of defocus-based depth estimation integrates thin-lens physics into novel 3D representations, such as 3D Gaussian Splatting (3DGS). Standard 3DGS algorithms assume an idealized pinhole camera model. When these models are trained on real-world images containing out-of-focus regions, the optimization process struggles, often generating floating artifacts ("floaters") or corrupting the underlying 3D geometry to mimic the 2D blur.
By replacing the pinhole assumption with a finite-aperture model, modern frameworks turn defocus blur into an optimization constraint. Zhao et al. introduced a differentiable defocus convolution that models the Circle of Confusion based on the depth of each 3D Gaussian primitive. This provides a physical gradient signal during training: if a rendered primitive's blur does not match the blur in the ground-truth image, the optical mismatch generates a gradient that pushes the primitive toward its correct geometric position in 3D space.
[3D Gaussian Primitives]
|
v
[Finite-Aperture Rendering] ---> Calculates Circle of Confusion (CoC) per Primitive
|
v
[Rendered Defocus Blur] <--- (Optical Mismatch Gradient) ---> [Ground-Truth Blurry Image]
|
v
[Pushes Primitives to Correct 3D Depth]
Similarly, Lee et al. developed CoCoGaussian, which physically incorporates the Circle of Confusion within the 3DGS pipeline to recover sharp scenes from highly defocused inputs. By grounding the blur model within a 3D spatial representation rather than applying 2D image-space deblurring, these methods—along with parallel work by Wang et al. and Peng et al.—successfully decouple true scene texture from depth-dependent optical blur. This enables both high-fidelity 3D reconstruction from imperfect images and precise, post-capture cinematic refocusing control.
Key Foundational Work
A New Sense for Depth of Field (1985) — contributed 2 concept elements to this direction.
Minimal operator set for passive depth from defocus (1996) — contributed 1 concept elements to this direction.