EM Generalist: A physics-driven diffusion foundation model for electron microscopy

He Sun, Enze Ye, Chixiang Lu, Zixuan Jiang, Weimin Bai, Shougang Ren, Chenyu Wang, Jinyang Zhang, Ruohua Shi, Лей Ма, Kun Song, Xiaojuan Qi, Haibo JiangResearch Square 2025 · DOI

TL;DR

This paper introduces "EM Generalist," a novel diffusion-based foundation model that can perform various electron microscopy (EM) image restoration tasks (denoising, super-resolution, deblurring, 3D reconstruction) without requiring paired training data. By leveraging a massive, diverse dataset and incorporating physics-informed image degradation models, EM Generalist achieves state-of-the-art performance and broad generalization capabilities, making high-quality EM analysis more accessible.

What Problem Does This Paper Address?

Electron microscopy (EM) is crucial for visualizing biological structures at the nanoscale, but obtaining high-quality images, especially 3D volumes, is often time-consuming and technically challenging. Existing deep learning methods for EM image restoration typically require large, paired datasets (corrupted image + ground truth) for training. This is a significant bottleneck because acquiring such paired data is labor-intensive and costly. Furthermore, these supervised methods often struggle to generalize to new imaging conditions or sample types (out-of-distribution data), limiting their practical applicability. The paper aims to overcome these limitations by developing a more flexible, generalizable, and data-efficient approach.

Methods

The core of EM Generalist is a diffusion-based foundation model. Diffusion models work by learning to reverse a process of gradually adding noise to an image until it becomes pure noise. By learning this reverse process, the model can then generate clean images from noise.

Key innovations include:

  • Physics-Driven Bayesian Inversion Framework: Instead of just learning from data, EM Generalist integrates physical models of image degradation (e.g., noise, blurring, downsampling) into a Bayesian inversion framework. This allows the model to leverage prior knowledge about how images are corrupted.
  • Zero-Shot Reconstruction: The model is trained once on a large, unpaired dataset. At inference time, it can perform various restoration tasks without needing task-specific fine-tuning or paired data. This is achieved by simply swapping out the physics-based degradation model for the specific task.
  • Expectation-Maximization (EM) for Unknown Parameters: For scenarios where the exact degradation model is unknown (e.g., unknown blur kernels), an iterative EM algorithm is used to simultaneously estimate the model parameters and reconstruct the image.

Datasets

  • Training Data: The foundation model was trained on a massive dataset of over 1.7 million high-quality EM and vEM image patches (128x128 pixels) sourced from the internet. This dataset spans diverse cellular and tissue types and morphologies. Notably, only a small fraction (1.3%) of these patches contained paired corrupted and ground-truth images, highlighting the model's ability to learn from unpaired data.
  • Evaluation Datasets: For specific tasks like denoising, the authors acquired new, paired clean and degraded images from five biological specimens (stigma cells of Arabidopsis thaliana, mouse kidney, HeLa cells, mouse heart, mouse oocytes). For other tasks like super-resolution and deblurring, they used mouse brain cortex samples and ultra-large FOV images. For 3D reconstruction, they used datasets from mouse liver, mouse kidney, mouse brain, and fly brain.

Key Insights and Ideas

  • Unpaired Data is Sufficient: The most significant insight is that a powerful generative prior for EM images can be learned from vast amounts of unpaired data, eliminating the need for costly paired datasets for many restoration tasks.
  • Physics + Data is Powerful: Integrating physics-based degradation models into a diffusion framework allows for more robust and generalizable reconstructions, especially when dealing with complex or unknown degradations.
  • "Generalist" Approach: The ability to perform multiple tasks (denoising, super-resolution, deblurring, 3D reconstruction) with a single, pre-trained model is a major step towards a unified solution for EM image processing.
  • Improved Downstream Analysis: The high-quality reconstructions not only look better but also directly facilitate downstream analyses like connectomics and segmentation, reducing manual effort and improving accuracy.

Key Results

  • Denoising: EM Generalist achieved superior denoising performance compared to supervised methods like EMDiffuse, outperforming it in over 97% of samples, especially on out-of-distribution data.
  • Super-Resolution: Achieved a 32% improvement in resolution for 4-fold down-sampled images.
  • Defocus Correction: Improved resolution by 30% in SEM images.
  • 3D Isotropic Reconstruction: Successfully reconstructed isotropic volumes from anisotropic vEM data, recovering lost synaptic vesicles and improving segmentation accuracy by approximately 40% in connectome analyses.
  • Reduced Errors: In connectomic analysis, the false negative rate for dendritic spine identification decreased from 1.81% to 0.45%, and over-segmentation errors per spine decreased by 30-40%.

Limitations and Future Work

  • Computational Demands: Diffusion models are iterative, leading to significant computational costs. Denoising a single large 2D image takes about 4 minutes, and 3D reconstruction takes about 1.5 minutes on an NVIDIA A800 GPU.
  • Future Work: The authors plan to focus on accelerating inference through techniques like diffusion distillation to achieve near real-time restoration. They also envision continued development to enhance automated, high-throughput ultrastructural analysis and promote reproducibility.

Summary updated

Abstract

<title>Abstract</title> Electron microscopy (EM) is an indispensable tool for visualizing biological structures at the nanoscale. However, high-fidelity EM imaging is often time-consuming, and 3D isotropic volume EM (vEM) remains impractical for large-scale analysis. Recent supervised deep learning approaches have partially mitigated these limitations, but they rely heavily on large paired datasets for training and struggle to generalize to out-of-distribution samples or new tasks. Here, we introduce EM Generalist, a diffusion-based generative foundation model trained on over 1.7 million high-quality EM samples from Internet-scale public datasets, and is coupled with a Bayesian inversion framework that incorporates physics-driven image degradation models during sampling to enable generalization across diverse data distributions and tasks. We demonstrated that EM Generalist is capable of addressing various tasks, including denoising, super-resolution, defocus correction, and 3D isotropic volume reconstruction without requiring paired training data for these tasks. This capability enables high-quality, zero-shot reconstructions across various imaging scenarios and diverse sample types.
BibTeX
@inproceedings{sun2025em,
  author = {He Sun and Enze Ye and Chixiang Lu and Zixuan Jiang and Weimin Bai and Shougang Ren and Chenyu Wang and Jinyang Zhang and Ruohua Shi and Лей Ма and Kun Song and Xiaojuan Qi and Haibo Jiang},
  title = {{EM Generalist: A physics-driven diffusion foundation model for electron microscopy}},
  booktitle = {Research Square},
  year = {2025},
  doi = {10.21203/rs.3.rs-6602548/v1},
}
Figure