Fourier ptychographic microscopy image stack reconstruction using implicit neural representations

Haowen Zhou, Brandon Y. Feng, Haiyun Guo, Siyu Lin, Mingshu Liang, Christopher A. Metzler, Changhuei Yang

TL;DR

This paper introduces FPM-INR, a novel framework that combines physics-based optical models with implicit neural representations (INRs) to reconstruct high-resolution 3D image stacks from Fourier ptychographic microscopy (FPM) data. FPM-INR significantly accelerates reconstruction speed and drastically reduces memory requirements compared to traditional FPM methods, making it more practical for applications like digital pathology.

What Problem Does This Paper Address?

Fourier ptychographic microscopy (FPM) is a powerful technique for acquiring high-resolution, wide field-of-view 3D image stacks without the need for physical z-stack scanning. This is a significant advantage for applications like digital pathology, where rapid acquisition of large volumes of data is crucial. However, conventional FPM reconstruction methods are computationally intensive, taking tens of minutes to process and requiring gigabytes of storage for a single high-resolution volume. This bottleneck hinders real-time applications and efficient data handling. While deep learning approaches have been explored, they often suffer from poor generalization to new datasets and can produce unreliable artifacts. This paper aims to bridge this gap by developing a FPM reconstruction framework that is fast, memory-efficient, and robust, while still adhering to the underlying physics of the imaging process.

Methods

The core innovation of FPM-INR lies in its use of Implicit Neural Representations (INRs) to represent the 3D image stack. Instead of storing a discrete 3D grid of voxels, FPM-INR learns a compact representation within a neural network. Specifically, it employs a low-rank decomposition of the volumetric data into a full-rank matrix (M) for lateral (x-y) variations and a 1D vector (u) for axial (z) variations. A multi-layer perceptron (MLP) then maps these feature vectors to the complex optical field at any given (x, y, z) coordinate.

The reconstruction process is self-supervised, using the FPM forward model (which describes how light propagates from the sample to the camera) to compare estimated measurements with the actual raw data. The framework iteratively updates the weights of the MLP and the parameters of the feature space (M and u) through gradient descent. A key aspect of their optimization strategy is an alternating z-slice selection approach, which involves both uniformly spaced and randomly selected z-planes to efficiently train the model. For inference, FPM-INR supports continuous sampling along the z-axis, allowing for arbitrary depth refocusing.

Datasets

The paper evaluates FPM-INR using two primary datasets:

  • Human blood smear slide: This was used as a proof of concept. The data was acquired with a specific microscope setup and illumination scheme.
  • Thyroid gland lesion pap smear specimen: This dataset was used to demonstrate the application of FPM-INR in digital pathology.

The paper does not introduce new benchmark datasets but rather uses these real-world biological samples to validate the performance of their method against conventional FPM.

Key Insights and Ideas

  • INR for Volumetric Imaging: Applying INRs to FPM reconstruction is a novel approach that allows for a compact and continuous representation of 3D volumetric data, overcoming the limitations of discrete voxel storage.
  • Physics-Informed Representation: FPM-INR is not a purely data-driven black box. It integrates the physics of the FPM forward model into the reconstruction process, ensuring that the learned representations are physically plausible and less prone to hallucinations.
  • Feature Space Design: The use of a low-rank decomposition (matrix M and vector u) for the feature space is crucial for efficiently representing volumetric data while accounting for the anisotropic resolution of optical microscopy.
  • Self-Supervised Learning with FPM Forward Model: The framework leverages the known physics of FPM to create a self-supervised learning loop, eliminating the need for external ground truth data or pre-training on specific datasets.
  • Adaptability: FPM-INR is designed to be agnostic to system design and sample types, requiring no hardware modifications and no pre-training, making it broadly applicable.

Key Results

  • Speedup: FPM-INR achieves a significant acceleration in reconstruction speed, outperforming traditional FPM algorithms by up to 25x in their experiments.
  • Memory Reduction: The framework drastically reduces memory requirements, compressing FPM z-stack data by up to 80x compared to conventional methods. For a human blood smear sample, the reconstructed data volume was reduced from 2576 MB to 32 MB.
  • Image Quality: While the primary goal was not to improve image quality, FPM-INR generally produced comparable or slightly better results than conventional FPM, with a notable reduction in artifacts, especially at larger defocus distances.
  • Continuous Inference: The method supports continuous refocusing along the z-axis, enabling seamless visualization of volumetric data.

Limitations and Future Work

  • Limitations:
    • The paper mentions that while FPM-INR generally achieves good quality, the primary goal was not to surpass conventional FPM in image quality but rather to improve speed and memory efficiency.
  • The performance comparison for speedup and compression ratio is presented for amplitude images only; including phase images would double the data volume and storage size for both FPM and FPM-INR.
  • Future Work:
    • The authors suggest that FPM-INR can be further streamlined for downstream tasks, implying potential for integration into automated analysis pipelines.
  • The adaptability of FPM-INR to various FPM setups suggests its potential for broader applications in fields beyond biological imaging.
  • The paper hints at further applications in critical areas like healthcare, suggesting future research directions in clinical diagnosis and disease analysis.

Summary updated

Abstract

Image stacks provide invaluable 3D information in various biological and pathological imaging applications. Fourier ptychographic microscopy (FPM) enables reconstructing high-resolution, wide field-of-view image stacks without z -stack scanning, thus significantly accelerating image acquisition. However, existing FPM methods take tens of minutes to reconstruct and gigabytes of memory to store a high-resolution volumetric scene, impeding fast gigapixel-scale remote digital pathology. While deep learning approaches have been explored to address this challenge, existing methods poorly generalize to novel datasets and can produce unreliable hallucinations. This work presents FPM-INR, a compact and efficient framework that integrates physics-based optical models with implicit neural representations (INRs) to represent and reconstruct FPM image stacks. FPM-INR is agnostic to system design or sample types and does not require external training data. In our experiments, FPM-INR substantially outperforms traditional FPM algorithms with up to a 25-fold increase in speed and an 80-fold reduction in memory usage for continuous image stack representations.
BibTeX
@inproceedings{zhou2023fourier,
  author = {Haowen Zhou and Brandon Y. Feng and Haiyun Guo and Siyu Lin and Mingshu Liang and Christopher A. Metzler and Changhuei Yang},
  title = {{Fourier ptychographic microscopy image stack reconstruction using implicit neural representations}},
  booktitle = {Optica},
  year = {2023},
  doi = {10.1364/optica.505283},
}
Figure