UniEM-3M: A Universal Electron Micrograph Dataset for Microstructural Segmentation and Generation

Nan wang, Zhiyi Xia, Yiming Li, Shi Tang, Zuxin Fan, Xi Fang, Haoyi Tao, Xiaochen Cai, Guolin Ke, Linfeng Zhang, Yanhui Hong

Abstract

Quantitative microstructural characterization is fundamental to materials science, where electron micrograph (EM) provides indispensable high-resolution insights. However, progress in deep learning-based EM characterization has been hampered by the scarcity of large-scale, diverse, and expert-annotated datasets, due to acquisition costs, privacy concerns, and annotation complexity. To address this issue, we introduce UniEM-3M, the first large-scale and multimodal EM dataset for instance-level understanding. It comprises 5,091 high-resolution EMs, about 3 million instance segmentation labels, and image-level attribute-disentangled textual descriptions. Furthermore, we release a text-to-image diffusion model trained on the collection to serve as a data augmentation tool. To establish a rigorous benchmark, we evaluate various instance segmentation methods and present UniEM-Net as a strong baseline model. Quantitative experiments demonstrate that this flow-based model outperforms other advanced methods on this challenging benchmark.

Introduction

Quantitative microstructural analysis—measuring the size, shape, and distribution of particles or grains—is the backbone of materials science. While Electron Microscopy (SEM and TEM) provides necessary high-resolution imagery, analyzing these images is traditionally a manual, labor-intensive process. Deep learning offers a solution, but it has been hindered by a "data desert." Existing EM datasets are typically small, lack pixel-level instance labels, or are heavily biased toward biological samples. Material microstructures are incredibly diverse and often "crowded," with a single image containing thousands of overlapping instances, causing standard AI models designed for natural images to fail.

The UniEM-3M Dataset

UniEM-3M is a massive new multimodal dataset featuring over 5,000 high-resolution EMs and 3 million expert-annotated instance labels. It is significantly larger and more "dense" than previous benchmarks like EMPS or MMSci.

Overview of the UniEM-3M dataset, showcasing raw EMs, segmentation labels, and structured textual descriptions.

A key feature of UniEM-3M is its multimodal data engine, which generates structured, nine-dimension textual descriptions for every image (covering material type, morphology, density, etc.). This creates a "language" for microstructures that allows for precise control over image generation and more interpretable AI analysis.

Methodology

The authors addressed the challenges of microstructural analysis through two primary innovations:

  1. UniEM-Net: Instead of using traditional "anchor-based" methods (which draw bounding boxes), UniEM-Net is a flow-based model. It learns to predict "directional fields" that point every pixel toward the center of its respective object, which is far more effective for separating thousands of tightly packed particles.
  2. Generative Augmentation (SDXL + LoRA): The researchers fine-tuned the Stable Diffusion XL model using Low-Rank Adaptation (LoRA) on UniEM-3M. This allows for the generation of high-fidelity synthetic micrographs to serve as a proxy for the complete data distribution, especially useful where privacy concerns limit the release of original proprietary samples.

Key Results

UniEM-Net achieved a mean Average Precision ([email protected]) of 0.824 on sparse data and 0.787 on dense data, significantly outperforming established models like Mask R-CNN (0.542) and specialized tools like Cellpose-SAM (0.760).

Visual comparison of segmentation performance across different models, highlighting UniEM-Net's superiority in dense microstructures.

One of the most interesting findings is that a model pre-trained on 30,000 generated (synthetic) images performed better on real-world tasks than a model pre-trained on the massive SA1B natural image dataset. This suggests that domain-specific "style" and density characteristics are more important than sheer data volume. The generative model achieved a Fréchet Inception Distance (FID) score of 34.03, remarkably close to the real data score of 28.06.

Conclusion

UniEM-3M provides a comprehensive benchmark and a powerful generative tool for automated materials analysis. While currently dominated by SEM images, future work will expand coverage of other imaging modalities and explore deeper vision-language modeling to teach AI to reason about material properties based on visual structure.

Summary updated

Abstract

Quantitative microstructural characterization is fundamental to materials science, where electron micrograph (EM) provides indispensable high-resolution insights. However, progress in deep learning-based EM characterization has been hampered by the scarcity of large-scale, diverse, and expert-annotated datasets, due to acquisition costs, privacy concerns, and annotation complexity. To address this issue, we introduce UniEM-3M, the first large-scale and multimodal EM dataset for instance-level understanding. It comprises 5,091 high-resolution EMs, about 3 million instance segmentation labels, and image-level attribute-disentangled textual descriptions, a subset of which will be made publicly available. Furthermore, we are also releasing a text-to-image diffusion model trained on the entire collection to serve as both a powerful data augmentation tool and a proxy for the complete data distribution. To establish a rigorous benchmark, we evaluate various representative instance segmentation methods on the complete UniEM-3M and present UniEM-Net as a strong baseline model. Quantitative experiments demonstrate that this flow-based model outperforms other advanced methods on this challenging benchmark. Our multifaceted release of a partial dataset, a generative model, and a comprehensive benchmark -- available at huggingface -- will significantly accelerate progress in automated materials analysis.
BibTeX
@misc{wang2025uniem,
  author = {Nan wang and Zhiyi Xia and Yiming Li and Shi Tang and Zuxin Fan and Xi Fang and Haoyi Tao and Xiaochen Cai and Guolin Ke and Linfeng Zhang and Yanhui Hong},
  title = {{UniEM-3M: A Universal Electron Micrograph Dataset for Microstructural Segmentation and Generation}},
  year = {2025},
  eprint = {2508.16239},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.16239},
}
Figure