When a camera lens focuses on a specific plane in a scene, light rays originating from a point on that plane converge perfectly to a single point on the image sensor, creating a sharp pixel. Points located closer to or farther from the lens do not converge on the sensor; instead, they project a cone of light that intersects the sensor plane as a blurry disc known as the circle of confusion.
In computational photography, Depth from Defocus (DfD) and Depth from Focus (DfF) techniques attempt to invert this process, treating the size and characteristics of this blur as a geometric signal to map the 3D structure of a scene. However, these techniques face a fundamental physical limitation: optical symmetry.
The Symmetry of Blur
The primary challenge of estimating depth from a single blurred image is that the size of the circle of confusion depends on the absolute distance from the focal plane, not the direction. An object placed a certain distance in front of the focal plane can produce an identical blur radius to an object placed at a corresponding distance behind it.
As Pentland demonstrated in the foundational work A New Sense for Depth of Field, this optical symmetry creates an inherent sign ambiguity. A standard circular aperture produces uniform, isotropic blur patterns that fail to distinguish between near and far defocus levels (Shedligeri et al.; Arguello et al.). Without modifying the capture process or introducing external constraints, a single-image defocus model cannot definitively determine whether an object is in front of or behind the point of perfect focus.
Object (Too Near) ───> \ / ───> Focuses BEHIND Sensor ───> [ Blur Disc ]
Focal Plane ───> | Lens | ───> Focuses EXACTLY on Sensor ──> [ Sharp Point ]
Object (Too Far) ───> / \ ───> Focuses IN FRONT of Sensor ─> [ Blur Disc ]
To resolve this ambiguity and establish a reliable relationship between blur and depth, researchers have developed three primary strategies: optical hardware engineering, sub-pixel phase analysis, and data-driven semantic priors.
Breaking Symmetry via Optical Engineering
One elegant way to resolve focal plane ambiguity is to modify the camera's optics so that the blur patterns themselves become asymmetric.
Coded Apertures
Standard lenses use circular apertures, which preserve the sign ambiguity because their point spread functions (PSFs) are symmetric on either side of the focal plane. By inserting a structured mask inside the lens—a technique known as a coded aperture—the light distribution is altered.
If the mask pattern is asymmetric or color-filtered, the resulting PSF changes its shape or color distribution depending on whether the light rays are converging or diverging when they hit the sensor. For example, Arguello et al. utilized a learned color-coded aperture to embed depth-specific color signatures directly into the blur patterns of a single image, allowing a computational decoder to immediately identify the direction of depth.
Focal Sweeping
Alternatively, the camera can actively alter its focus during or across exposures to sample the focal gradient. Nagahara et al. introduced "half-sweep imaging," which splits a continuous focus sweep into two distinct halves during a single exposure to optimize the camera's frequency response and isolate depth cues.
On consumer hardware, this principle is often applied by capturing a "focal stack"—a sequence of images taken at different focus distances. Suwajanakorn demonstrated that the rapid, automatic focus adjustments a smartphone makes when a user taps the screen can be captured as a focal stack and transformed into high-fidelity depth maps, bypassing single-frame ambiguity by tracking how blur changes across multiple known focal planes.
Phase and Disparity Cues at the Sensor Level
Instead of altering the lens, modern imaging systems frequently leverage specialized sensor architectures to capture the direction of light rays. Dual-Pixel (DP) sensors, which are standard in many modern smartphones and DSLR cameras, split each individual pixel into two independent left and right photodiodes beneath a shared micro-lens.
In-Focus Light Rays: Converge perfectly ──> Both photodiodes receive identical signals.
Out-of-Focus (Near/Far): Converge early/late ─> Left/Right photodiodes receive shifted signals.
This architecture creates a tiny sub-pixel phase shift, or defocus disparity, between the left and right sub-images. Crucially, the direction of this sub-pixel shift flips depending on whether the object is located in front of or behind the current focal plane.
The authors of Reflection Removal Using a Dual-Pixel Sensor exploited this property to separate overlapping image layers, using the phase shift direction to distinguish between an in-focus background and an out-of-focus reflection (Punnappurath). Furthermore, Ashida et al. demonstrated that mathematically coupling these dual-pixel defocus cues with standard multi-view geometry provides enough physical constraints to resolve the absolute scale of a 3D reconstruction without requiring external camera calibration.
Resolving Ambiguity with Semantic Priors
When hardware modifications or phase-detecting sensors are unavailable, resolving focal plane ambiguity from a standard circular-aperture image requires contextual information. Human vision accomplishes this seamlessly because we understand the semantic structure of our environment; we know that the sky is distant, ground planes are continuous, and objects do not float arbitrarily in space.
Modern deep learning architectures mimic this capability by combining local blur measurements with global semantic understanding. Rather than relying solely on mathematical focus metrics, these networks are trained on large datasets to recognize scene layouts.
- Aperture Supervision: Srinivasan et al. trained monocular depth networks by comparing all-in-focus images with shallow depth-of-field images. The network learns to resolve the sign ambiguity by using the scene's semantic context to "explain away" the observed blur patterns.
- Differentiable Optics: Gur et al. embedded a differentiable PSF convolutional layer directly into a neural network's training loop. This forces the model to output structurally and physically plausible depth maps that can successfully re-render the original blurred image.
- Predictive Autofocus: In active camera systems, Herrmann et al. trained convolutional neural networks on focal stacks to instantly predict both the magnitude and the direction of focus error from a single out-of-focus frame, eliminating the traditional trial-and-error search routines that plague standard autofocus systems.
Real-World Complications and Failure Modes
While the geometric models underlying depth-from-defocus are mathematically sound, real-world scenes present several optical phenomena that complicate depth estimation:
- Information Leakage and Halos: When a foreground object is severely out of focus, its light bleeds into adjacent background pixels. Sakurikar et al. showed that this extreme defocus creates multiple focus peaks and halo artifacts, obscuring the true focal point and causing standard focus measures to fail near object boundaries.
- Specular Highlights: Saturated highlights in out-of-focus regions expand into high-contrast circular bokeh patterns. Traditional edge-detection focus measures often mistake the sharp boundaries of these bokeh circles for in-focus edges, leading to severe depth mapping errors (Jeon et al.).
- Geometric vs. Optical Conflation: In modern 3D scene optimization frameworks, such as 3D Gaussian Splatting, training on images with shallow depth-of-field introduces a unique optimization ambiguity. As Shen et al. observed, optimization algorithms frequently confuse optical defocus blur with geometric placement errors, incorrectly positioning 3D primitives in space to compensate for the blur.
Key Foundational Work
A New Sense for Depth of Field (1985) — contributed 2 concept elements to this direction.
Half-Sweep Imaging for Depth from Defocus (2011) — contributed 1 concept elements to this direction.