Microscopes and close-up cameras can keep only a thin slice of a scene sharp. To see a whole diatom shell, an insect leg or a macro subject, people capture a focal stack: many images of the same field, each focused at a different depth. Today the stack is usually collapsed into one "all-in-focus" picture by fusion software, and a segmentation model then runs on that picture. The fusion step fails in a predictable place. Where a sharp object meets a blurred background, out-of-focus light spills across the edge (the defocus spread effect). The resulting halos and smeared margins are fixed into the fused image before the segmenter ever sees it. We propose to segment directly from the stack with a coarse-to-fine model. A cheap low-resolution pass finds objects and estimates how blurred each boundary is. An expensive high-resolution pass then works only inside a band around each boundary. The width of that band, and which side of the edge it extends toward, come from the estimated optical blur rather than a fixed hand-set kernel.
The idea is specific and falsifiable. If a fixed-width band, tuned once on validation data, does as well as an optics-sized band, the physical modeling adds nothing and the proposal fails. If a good fuse-then-segment pipeline matches end-to-end stack segmentation, the coarse-to-fine machinery is unnecessary. Both outcomes are tested head-on below.
Core Claim
Three bodies of work each hold one piece of this, and none connects them. Multi-focus fusion research models boundary physics carefully but stops at producing a fused image. Multi-Focus Image Fusion via Explicit Defocus Blur Modelling (DMANet) estimates spatially varying blur kernels, meaning the shape of the blur at each pixel, and passes uncertain boundary pixels to a separate fuser. An α-Matte Boundary Defocus Model Based Cascaded Network for Multi-focus Image Fusion (MMF-Net) models scenes as layered surfaces with soft transparency masks. It shows that the spread is asymmetric: a blurred foreground bleeds outward over the background, while a blurred background stays behind a sharp foreground. Its boundary stage, however, marks the boundary with a fixed guidance value of 0.5 and works only on two-image pairs.
Coarse-to-fine segmentation research has the efficiency machinery but no optics. No More Sliding Window (NMSW) runs a global network on a downsampled CT volume. It then picks the K most informative high-resolution patches with a differentiable sampler and reports a 91% cut in compute while matching sliding-window Dice. CT and MRI scans, though, have no lens blur. Polyper splits a coarse mask into a confident interior and a "potential boundary" band using fixed erosion and dilation. It then sharpens the band with cross-attention from interior features. Its authors name the fixed band width as a limitation.
Our claim is that, in focal stacks, the right refinement region is predictable from the optics. Its width scales with the local blur radius, and it is offset toward the blurred side of the edge. Making the coarse stage output this region, and spending fine-stage compute only there, should improve boundary accuracy over fuse-then-segment pipelines and over fixed-band refinement at equal compute. The two papers that supply the method are NMSW (patch sampling) and Polyper (interior-to-boundary attention). The fusion papers supply the blur model. Real-MFF and MFFW provide the motivating evidence. Both report that deep networks trained on synthetic blur break down on complex real boundaries such as hair and foliage, where classical transform-domain fusion beats them.
Research Program
Coarse stage. The model takes the stack downsampled spatially, with all slices kept. It predicts three per-pixel outputs:
- a coarse object mask;
- a "best-focus slice index," acting as a cheap depth proxy;
- a blur radius $r(x)$, the estimated size of the blur disc at pixel $x$.
For $r(x)$ we reuse DMANet's idea of fitting a small dictionary of Gaussian blur shapes, applied per slice. From these outputs we build an asymmetric boundary band. Consider a mask boundary point $x$ where the depth proxy changes. The band extends a distance $\beta\, r_{\text{fg}}(x)$ outward into the background and only a small distance $\gamma$ inward. Here $r_{\text{fg}}(x)$ is the blur radius of the nearer layer in the slice where the farther layer is sharp, and $\beta, \gamma$ are two scalars learned on training data. This encodes MMF-Net's asymmetry in place of Polyper's symmetric morphology.
Fine stage. We then sample high-resolution patches using an NMSW-style differentiable top-K selector. Its importance map is the band area weighted by coarse-mask uncertainty. Before selection we suppress overlapping proposals, which addresses NMSW's known problem of oversampling around small targets.
Inside each patch, the fine network reads all original slices at full resolution, not a fused image. It uses Polyper-style attention from confident interior features to classify band pixels. This matters because of what MFFW calls the inpainting dilemma. Behind a blurred foreground edge, no single slice is clean. The relevant evidence is spread across several slices, and a fused image has already discarded it. As a secondary output, the fine stage can also emit a fused image inside the band, so the same system can be scored on fusion benchmarks.
Training. We first train the stages separately, then fine-tune them jointly through the differentiable sampler. Training data comes from three sources:
- focal stacks rendered from real light-field captures, following the Real-MFF procedure, which avoids the synthetic-Gaussian domain gap Real-MFF documents;
- MMF-Net-style layered composites with soft masks;
- real microscopy stacks, wherever raw slices are available.
Baselines
The naive reference is the pipeline used in UDE DIATOMS in the Wild 2024: commercial focus stacking (Helicon Focus), then a standard U-Net trained on the fused images. Every learned baseline gets the same training images, the same annotation set, and the same backbone family.
The strongest fuse-then-segment competitor replaces commercial stacking with the best physics-aware fusion we can run. That is DMANet on photographic data and U-Swin on microscopy, followed by the same U-Net. This is the real competitor. If fusion has already solved the boundary, this pipeline wins. For fusion quality itself, the reported numbers are as follows:
- DMANet: 42.898 dB PSNR on MFI-WHU.
- NSCT-SR: 39.85 dB PSNR on the Real-MFF complex subset.
- U-Swin: 43.85 dB PSNR on EDoF Fraunhofer.
For object segmentation on focal stacks, we found no reported result on Lacuna for any of these benchmarks, so we must establish segmentation numbers ourselves. To beat this competitor, our method must show higher boundary F-score and lower boundary distance error on held-out real stacks, at equal or lower inference compute, with confidence intervals across seeds that exclude zero.
The dense end-to-end competitor is a 3D sliding-window U-Net over the full raw stack, given the same parameter budget. It sees all the information our model sees. Matching its accuracy at a fraction of its compute (MACs and wall-clock) is the NMSW-style win condition.
The ablation baselines isolate the key ingredient by changing only the band:
- a symmetric fixed-width band, Polyper-style, with width tuned by grid search on validation data under the same budget we spend tuning $\beta, \gamma$;
- a band whose width is predicted by a free network head with no blur supervision;
- our symmetric band, which drops the asymmetry while keeping the blur sizing.
The optics claim survives only if the full asymmetric, blur-sized band beats all three on the boundary metrics.
Why not just…?
Why not just fuse with a strong modern method, then segment? This is the most serious threat. DMANet's 3 dB lead over DSIFT on MFI-WHU suggests fusion has improved a lot. However, fused-image PSNR is dominated by interior pixels. Real-MFF and MFFW both report that the failures concentrate at complex boundaries, which are exactly where segmentation errors live. The fuse-then-segment baseline above settles the question. We will also report errors stratified by band membership, so a win confined to the bands is distinguishable from a general advantage. If the gap at boundaries is under one point of boundary F-score, we will conclude fusion suffices.
Why not just widen Polyper's fixed band and tune it? A single tuned width may be enough when blur sizes in a dataset are narrow-ranged. Polyper itself suggests fixed widths are a weakness, but only on polyps, which is a different setting. We expect optics-sizing to help most when blur radius varies widely within one image, as in deep microscopy stacks. We will therefore bin test boundaries by estimated blur radius. The prediction is that the fixed band matches us in the median bin and loses in the extremes. If it matches everywhere, we drop the physics claim.
Why not just run a dense network on the whole stack? It is likely about as accurate, since it sees everything. NMSW shows the cost is large: 99.0 s versus 8.3 s per CT volume on an H100. Stacks of 40–85 high-resolution slices have a similar cost profile. Our argument here is about accuracy per unit of compute, not raw accuracy, and we will present accuracy-versus-compute curves rather than single points.
Proposed Experiments
Photographic boundaries. Real-MFF renders focal stacks from light fields, so object masks can be annotated on its all-in-focus ground truth. We will annotate masks for a subset of the complex split; the dataset does not supply them. We will report:
- boundary F-score at several tolerances, overall mIoU, and errors split by blur-radius bin;
- PSNR inside the band for the secondary fused output, compared against NSCT-SR's 39.85 dB on the same complex subset;
- results on MFFW's manually annotated focus maps, which test decision-map accuracy on real "in the wild" spread effects.
Microscopy, the target application. We will use UDE DIATOMS stacks, provided the raw 40–85 slice scans can be obtained. The file confirms they were acquired but not that they are released, and this is the main data risk. If they are unavailable, we will use EDoF Fraunhofer-style microscopy stacks with new mask annotations. Thin shell margins and appendages are where baked-in halos should hurt most.
Decisive checks. Two outcomes decide the project. The first is the boundary-stratified comparison against the DMANet/U-Swin fuse-then-segment pipeline. The second is the blur-binned comparison against a tuned fixed band. If both favor the proposed method at equal compute, the claim holds. If only the compute-efficiency result holds, the contribution narrows to NMSW-style sampling for optical stacks, and we will report it as such.
References
- Multi-Focus Image Fusion via Explicit Defocus Blur Modelling
- An α-Matte Boundary Defocus Model Based Cascaded Network for Multi-focus Image Fusion
- No More Sliding Window: Efficient 3D Medical Image Segmentation with Differentiable Top-K Patch Sampling
- Polyper: Boundary Sensitive Polyp Segmentation
- Real-MFF: A large realistic multi-focus image dataset with ground truth
- MFFW: A new dataset for multi-focus image fusion
- “UDE DIATOMS in the Wild 2024”: a new image dataset of freshwater diatoms for training deep learning models
- Multi-Focus Microscopy Image Fusion Based on Swin Transformer Architecture