Unsupervised content-preserving transformation for optical microscopy

Xinyang Li, Guoxun Zhang, Hui Qiao, Feng Bao, Yue Deng, Jiamin Wu, Yangfan He, Jing‐Ping Yun, Xing Lin, Hao Xie, Haoqian Wang, Qionghai DaibioRxiv 2019 · DOI

Abstract

The development of deep learning and the open access to a substantial collection of imaging data provide a potential solution to computational image transformation, which is gradually changing the landscape of optical imaging and biomedical research. However, current implementations of deep learning usually operate in a supervised manner and the reliance on a laborious and error-prone data annotation procedure remains a barrier towards more general applicability. Here, we propose an unsupervised image transformation to facilitate the utilization of deep learning for optical microscopy, even in some cases where supervised models cannot apply. By introducing a saliency constraint, the unsupervised model, dubbed as Unsupervised content-preserving Transformation for Optical Microscopy (UTOM), can learn the mapping between two image domains without requiring paired training data and avoid the distortion of the image content. UTOM shows promising performances in a wide range of biomedical image transformation tasks, including in silico histological staining, fluorescence image restoration, and virtual fluorescence labeling. Quantitative evaluations elucidate that UTOM achieves stable and high-fidelity image transformations across different imaging conditions and modalities. We anticipate that our framework will encourage a paradigm shift in training neural networks and enable more applications of artificial intelligence in biomedical imaging.

Introduction: The Registration Bottleneck

In biomedical imaging, the ability to transform one image modality into another—such as converting unstained tissue scans into virtual pathology slides or denoising low-light fluorescence—is transformative. Traditionally, these tasks rely on supervised learning, which requires thousands of perfectly "registered" image pairs. In these pairs, every pixel in the input must align exactly with the output.

Achieving this level of precision is a significant logistical challenge. Biological samples can shift, chemical staining processes often physically distort tissue, and many imaging modalities are simply incompatible for simultaneous capture. This "registration bottleneck" has historically prevented deep learning from being applied to many dynamic biological processes or rare clinical samples where paired data is impossible to obtain.

The UTOM Framework

To overcome these limitations, researchers developed UTOM (Unsupervised content-preserving Transformation for Optical Microscopy). Built upon the foundation of CycleGAN—an architecture designed to learn mappings between two domains (Domain A and Domain B) without paired examples—UTOM introduces a critical innovation to prevent the "hallucinations" or structural distortions common in standard unsupervised models.

The Saliency Constraint ($L_{sc}$)

The core of UTOM is the Saliency Constraint. Recognizing that microscopy images typically feature predictable backgrounds (often uniform black or white), the researchers introduced a mathematical constraint that extracts a "saliency map"—a mask identifying important biological structures—from both the input and the generated output.

  • Thresholding: The model uses a sigmoid-approximated threshold to distinguish biological "content" from the "background."
  • Consistency: A specific loss function forces the generator to ensure that the shape and position of these structures remain consistent throughout the transformation.
  • Dynamic Weighting: To balance detail and structure, the saliency constraint is weighted heavily at the start of training to guide the model toward the correct morphology, gradually decaying to allow the cycle-consistency loss to refine fine-grained textures.

Experimental Validation

UTOM was validated across a diverse array of biological datasets, demonstrating its versatility across different species and imaging tasks:

  1. In Silico Histological Staining: Using human colorectal tissue, UTOM learned to generate H&E-stained images from label-free sections. Notably, the model was trained using adjacent tissue sections—similar but not identical—proving it can function without pixel-perfect alignment.
  2. Fluorescence Image Restoration: In Planaria (flatworm) imaging, UTOM improved the Peak Signal-to-Noise Ratio (PSNR) by approximately 8 dB, resolving structures that were previously obscured by noise.
  3. Axial Resolution Recovery: In mouse liver samples, the model addressed Z-axis blur, improving PSNR by 5 dB and successfully separating structures that appeared merged in the original scans.
  4. Virtual Fluorescence Labeling: UTOM transformed phase-contrast images of human motor neurons into virtual fluorescence labels (identifying axons, dendrites, and nuclei) with a Structural Similarity Index (SSIM) of 0.88, approaching the performance of supervised models (typically 0.90+).

Key Insights and Future Directions

UTOM demonstrates that unsupervised learning does not have to be unguided. By leveraging the physical realities of microscopy—specifically the consistency of backgrounds—the framework achieves supervised-level accuracy without the associated data burden. This focus on content preservation is vital for medical diagnostics, where the semantic integrity of a cell or gland is more important than mere visual appeal.

While UTOM represents a major leap forward, some gaps remain. Supervised models still maintain a slight advantage in predicting complex intensity distributions and global lighting effects because they possess a pixel-aligned "ground truth." Future iterations may bridge this gap by incorporating physics-informed priors—models of light propagation—to further refine accuracy and reduce the number of parameters required. Ultimately, UTOM opens the door to applying AI to live-cell imaging and other high-speed dynamics where supervised registration will likely never be feasible.

Summary updated

Abstract

The development of deep learning and the open access to a substantial collection of imaging data provide a potential solution to computational image transformation, which is gradually changing the landscape of optical imaging and biomedical research. However, current implementations of deep learning usually operate in a supervised manner and the reliance on a laborious and error-prone data annotation procedure remains a barrier towards more general applicability. Here, we propose an unsupervised image transformation to facilitate the utilization of deep learning for optical microscopy, even in some cases where supervised models cannot apply. By introducing a saliency constraint, the unsupervised model, dubbed as Unsupervised content-preserving Transformation for Optical Microscopy (UTOM), can learn the mapping between two image domains without requiring paired training data and avoid the distortion of the image content. UTOM shows promising performances in a wide range of biomedical image transformation tasks, including in silico histological staining, fluorescence image restoration, and virtual fluorescence labeling. Quantitative evaluations elucidate that UTOM achieves stable and high-fidelity image transformations across different imaging conditions and modalities. We anticipate that our framework will encourage a paradigm shift in training neural networks and enable more applications of artificial intelligence in biomedical imaging.
BibTeX
@misc{li2019unsupervised,
  author = {Xinyang Li and Guoxun Zhang and Hui Qiao and Feng Bao and Yue Deng and Jiamin Wu and Yangfan He and Jing‐Ping Yun and Xing Lin and Hao Xie and Haoqian Wang and Qionghai Dai},
  title = {{Unsupervised content-preserving transformation for optical microscopy}},
  year = {2019},
  doi = {10.1101/848077},
  howpublished = {bioRxiv},
}
Figure