Multi-Focus Images Fusion for Fluorescence Imaging Based on Local Maximum Luminosity and Intensity Variance

Hao Cheng, Kaijie Wu, Chaochen Gu, Dingrui MaSensors 2024 · DOI

TL;DR

This paper introduces a novel multi-focus image fusion method for fluorescence microscopy that overcomes limitations of existing techniques by combining local maximum luminosity and intensity variance for sharper images. Crucially, it also enables the reconstruction of 3D surface models from these fused images, offering a more comprehensive view of biological samples.

What Problem Does This Paper Address?

High-resolution fluorescence microscopy, while powerful, is limited by a shallow depth of field. This means that in a single image, only a thin slice of the sample is in focus, leaving other parts blurred. Capturing a complete, in-focus image of thicker samples, like tissue slices, requires acquiring multiple images at different focal planes. Existing methods for fusing these "multi-focus" images often suffer from artifacts like blocking effects or the persistence of out-of-focus blur, and they typically don't provide depth information. This lack of depth information limits the ability to create 3D reconstructions, which are vital for understanding the spatial organization of biological structures.

Methods

The proposed method tackles these challenges through a multi-stage process:

  1. Image Collection and Block Segmentation: Multiple fluorescence images of the same sample area are captured at varying focal planes. The entire image area is then divided into non-overlapping rectangular blocks.
  2. Sharpness Evaluation and Depth Map Construction: A novel sharpness evaluation function is introduced. It combines local maximum luminosity (the brightest pixel value within a block) with intensity variance (how much pixel values vary within a block). This combined metric is more robust for fluorescence images than traditional sharpness measures. The block with the highest sharpness score in a given focal plane is used to estimate the depth for that block. A "confidence map" is also generated, indicating how reliable the depth estimation is for each block.
  3. Depth Adjustment: To ensure continuity and mitigate the impact of out-of-focus fluorescence from neighboring regions, the estimated depths of adjacent blocks are adjusted based on their confidence scores and proximity.
  4. Information Filling: To address blocking artifacts introduced by the block-based approach, an "information filling" method is employed. This step refines the depth estimation for individual pixels, especially at block boundaries, by considering neighboring block depths and pixel intensity values. This results in a detailed depth map for every pixel.
  5. Optional Step: Object Detection and Region Expansion: For samples with separable objects (like individual cells or structures), an optional step can be applied. This involves detecting these objects, adjusting block sizes based on object scale, and fusing images for each object class separately. This can improve efficiency and accuracy for complex samples.

The core novelty lies in the combined sharpness metric and the information-filling approach for depth estimation, which directly leads to the 3D reconstruction capability.

Datasets

The paper evaluates its method using two sets of fluorescence images:

  • Set 1: 81 fluorescence images of pollen grains.
  • Set 2: 60 fluorescence images of a spinach stem.

These datasets were acquired using an automatic fluorescence scanner. The paper does not introduce new benchmark datasets but rather uses these to compare its method against existing ones.

Key Insights and Ideas

  • Luminosity-Variance Synergy: The paper demonstrates that combining local maximum luminosity with intensity variance creates a superior sharpness metric for fluorescence microscopy compared to using either metric alone or traditional methods. This is because fluorescence intensity often correlates with focus, and variance captures textural detail.
  • Depth from Focus: The method ingeniously leverages the multi-focus image stack not just for a sharp 2D image but also to infer depth information for each pixel. This is a significant advantage over many existing fusion techniques.
  • Alleviating Blocking Artifacts: The information-filling step is crucial for producing visually seamless fused images and a continuous depth map, overcoming a common drawback of block-based processing.
  • 3D Reconstruction Potential: The generated depth map opens the door to reconstructing 3D surface models of biological samples, offering a richer understanding of their morphology and spatial relationships.

Key Results

  • State-of-the-Art Fusion: The proposed method achieves state-of-the-art performance in fusing multi-focus fluorescence images, producing sharper and more continuous results than compared methods.
  • Reduced Artifacts: The method significantly alleviates blocking effects and is less susceptible to out-of-focus fluorescence contamination compared to other approaches.
  • Computational Efficiency: Without the optional step, the proposed method is notably faster than several other advanced fusion techniques (e.g., SSSDI, GFF, De's method), making it practical for high-throughput applications.
  • 3D Depth Map Generation: The method successfully generates a depth map, enabling the reconstruction of 3D point clouds of observed objects, as demonstrated with a pollen grain.

Limitations and Future Work

  • Equipment Limitations: The authors acknowledge that they could not collect ground-truth depth images for quantitative analysis due to equipment limitations.
  • Generalizability: While the method shows promise, further experiments are needed to verify its applicability to a wider range of biological samples and scenarios.
  • Optional Step Complexity: The optional object detection and segmentation step, while beneficial for separable objects, can significantly increase computation time.
  • Parameter Tuning: The performance relies on carefully tuned parameters (thresholds, kernel sizes, ratios), which might require optimization for different imaging conditions.

Future work could focus on developing more automated parameter selection, exploring its application to different microscopy modalities, and further refining the 3D reconstruction capabilities.

Summary updated

Abstract

Due to the limitations on the depth of field of high-resolution fluorescence microscope, it is difficult to obtain an image with all objects in focus. The existing image fusion methods suffer from blocking effects or out-of-focus fluorescence. The proposed multi-focus image fusion method based on local maximum luminosity, intensity variance and the information filling method can reconstruct the all-in-focus image. Moreover, the depth of tissue's surface can be estimated to reconstruct the 3D surface model.
BibTeX
@inproceedings{cheng2024multi,
  author = {Hao Cheng and Kaijie Wu and Chaochen Gu and Dingrui Ma},
  title = {{Multi-Focus Images Fusion for Fluorescence Imaging Based on Local Maximum Luminosity and Intensity Variance}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24154909},
}
Figure