Multi-focus image fusion (MFIF) aims to create a single "all-in-focus" image by combining multiple shots of the same scene taken with different focal settings. While identifying sharp versus blurry regions is relatively straightforward in the center of an object, the process consistently breaks down at the edges. Recent research suggests that the primary obstacle to perfect fusion is a physical phenomenon known as the Defocus Spread Effect (DSE).
The Physics of Boundary Blur
In an ideal mathematical model, an image is divided into perfectly sharp regions and perfectly blurry ones. However, real-world optical lenses do not produce such clean breaks. As Junmin Liu and colleagues demonstrate in their work on the MFFW dataset, out-of-focus objects physically "expand" in the image. This expansion causes the blurred foreground to bleed into the sharp background, or vice versa.
This "bleeding" creates a transition zone where the pixel information is a complex mixture of focused and defocused light. Traditional algorithms that assume a static, sharp boundary between focus levels inevitably produce artifacts. As noted by Boyuan Ma in the development of SESF-Fuse, these failures manifest as ringing, serrated edges, or unnatural "halos" around objects.
Beyond Binary Masks
Early fusion methods often relied on "focus maps"—binary or near-binary masks that decide which source image a pixel should come from. But because of DSE, a pixel at an object's edge might not be "correct" in either source image; it might be contaminated by the spread of blur from a neighboring depth plane.
To solve this, researchers have moved toward explicit physical modeling:
- Alpha Matting: Haoyu Ma and Qingmin Liao proposed an $\alpha$-matte boundary defocus model. By treating the transition zone as a transparency problem (similar to how hair is cropped in photo editing), they can more accurately blend the light from different focal planes.
- Blur Kernel Estimation: Instead of just detecting sharpness, Yuhui Quan and Hui Ji introduced DMANet, which explicitly estimates spatially-varying blur kernels. This allows the network to "undo" the defocus spread before merging the images.
- Mask Expansion: In MFIF-GAN, Yicheng Wang and Shuang Xu found that slightly expanding the foreground focus mask beyond the physical boundary of the object helps capture the background regions affected by DSE, preventing ghosting artifacts.
Optimization and Unsupervised Learning
Because labeled real-world data for MFIF is difficult to obtain (you cannot easily take a "perfect" ground truth photo of a complex scene), the field is shifting toward unsupervised and optimization-based strategies.
Shuang Xu proposed an optimization-based framework that maximizes a novel structural similarity metric (MFF-SSIM). Rather than relying on a single pass of a neural network, this method uses gradient ascent to iteratively refine the fused image until the DSE halos disappear. Similarly, Fayez Lahoud and Sabine Süsstrunk demonstrated that zero-learning approaches can leverage pre-trained features to identify these boundaries without task-specific training, though they acknowledge that edge-preserving filters still struggle with the physical "expansion" of DSE.
Why It Works
The shift from "sharpness detection" to "DSE modeling" works because it acknowledges the camera as a physical instrument rather than a perfect mathematical sampler. By accounting for the fact that out-of-focus light physically occupies space reserved for focused light, these models can reconstruct the "hidden" sharp pixels at the boundaries. Whether through GANs, $\alpha$-matting, or iterative optimization, the goal is the same: to reconcile the overlapping boundaries created by the lens's aperture.