Every photographer and computer vision researcher eventually hits a physical wall: the depth-of-field (DOF) constraint. In a standard optical system, you can have a wide aperture (to let in more light) or a deep focus (to keep everything sharp), but rarely both. As noted by Murali Subbarao and colleagues in Focused image recovery from two defocused images recorded with different camera settings, this limitation results in distance-dependent blur that degrades downstream tasks like edge detection and 3D reconstruction.
To overcome this, researchers have developed Multi-Focus Image Fusion (MFF), often called "focus stacking." The goal is to take a "focal stack"—a sequence of images of the same scene captured at different focus distances—and merge them into a single, "all-in-focus" (AiF) composite.
The Intuition of Sharpness
The fundamental challenge of MFF is "focus measurement": how do we mathematically determine which pixels in which source image are the sharpest?
Early methods operated in the spatial domain. For instance, Kun Zhan and his team in A Novel Explicit Multi-focus Image Fusion Method utilized the Energy of Laplacian (EOL). The intuition here is that sharp edges contain high-frequency information; by calculating the second derivative (Laplacian) of the image, you can identify regions with the most "energy" or detail.
Other researchers looked to the transform domain. By converting images into frequencies, you can isolate the high-frequency components that represent focus. Bin Xiao et al. demonstrated in Multi-focus image fusion method based on discrete Tchebichef transform and focus measure that using low-order coefficients of the Tchebichef transform can provide a robust focus measure that is less sensitive to noise than simple spatial gradients.
Learning the Focus Mask
As deep learning matured, the field shifted from hand-crafted focus measures to learned ones. Instead of telling the computer to "look for high gradients," researchers began training neural networks to recognize focus directly.
A significant milestone was the development of unsupervised models, which don't require a perfect "ground truth" image for training. Xiang Yan and colleagues introduced MFNet, which uses structural similarity (SSIM) as a loss function to learn how to merge images. Similarly, Boyuan Ma proposed SESF-Fuse, which applies a "Spatial Frequency" metric directly to deep features extracted by an encoder-decoder network. These models essentially learn to generate a "decision map"—a binary or weighted mask that tells the system exactly which source image to trust for every pixel.
Solving the Boundary Problem
The hardest part of fusion isn't the middle of a focused object; it's the edge. When an object is out of focus, it "bleeds" into the background, a phenomenon known as the Defocus Spread Effect (DSE). If you simply stitch focused regions together, you often get blurry "halos" or artifacts at the boundaries.
To combat this, Yicheng Wang and Shuang Xu developed MFIF-GAN, a Generative Adversarial Network designed specifically to eliminate these boundary artifacts. By using a discriminator to judge whether the fused image looks "realistic" and sharp at the edges, the generator learns to handle the transition between focused and unfocused zones more gracefully. Haoyu Ma further refined this by modeling the boundary as an alpha-matte, treating the transition as a transparency problem rather than a hard cut.
Hardware and 3D Frontiers
While most MFF research focuses on post-processing 2D images, some work integrates the fusion process directly into the hardware or 3D reconstruction.
- Focal Sweep: Instead of taking discrete photos, Ryunosuke Yokoya and Shree K. Nayar explored Extended Depth of Field Catadioptric Imaging Using Focal Sweep. By moving the sensor during a single exposure, they create a controlled blur that can be mathematically deconvolved to produce a sharp image across a massive depth range.
- Neural Radiance Fields (NeRFs): In the cutting-edge realm of 3D scene synthesis, Xianrui Luo and colleagues developed DC-NeRF. This system uses a dual-camera setup on a smartphone to reconstruct a sharp 3D volume from blurry inputs, effectively performing multi-focus fusion in 3D space.
By moving from simple gradient-based stitching to deep, boundary-aware models and hardware-software co-design, this research direction has turned a fundamental optical limitation into a computational opportunity.