VesiclePy: A Machine Learning Vesicle Analysis Toolbox for Volume Electron Microscopy

Jason Ken Adhinarta, Yutian Fan, Adam Gohain, Michael Che-Hung Lin, Paige Nurkin, Richard Ren, Mark D. Roth, Shulin Zhang, Ayal Yakobe, Rafael Yuste, Donglai WeibioRxiv 2025 · DOI

TL;DR

This paper introduces VesiclePy, a novel machine learning pipeline for automatically segmenting, classifying, and analyzing vesicles in large-scale 3D electron microscopy datasets. It addresses the significant challenge of manually annotating these tiny, numerous cellular structures, enabling a more comprehensive understanding of neuronal communication.

What Problem Does This Paper Address?

Neurons communicate by releasing neurotransmitters packaged within vesicles. Understanding the types, numbers, and locations of these vesicles is crucial for deciphering neuronal function and communication mechanisms. However, analyzing vesicles in 3D volume electron microscopy (vEM) data is incredibly challenging. vEM provides nanoscale resolution but generates massive datasets containing tens of thousands of vesicles per neuron. Manually identifying, segmenting, and classifying each vesicle is a laborious, time-consuming, and error-prone process, hindering large-scale analysis. This paper aims to bridge this gap by providing an automated, efficient, and accurate tool for vesicle analysis in vEM.

Methods

VesiclePy is an integrated pipeline built on several key components:

  • Segmentation: For large vesicles, an iterative deep learning approach using a 3D U-Net architecture is employed, trained on user-annotated data. This is combined with human proofreading to refine predictions. Small vesicles are handled through manual segmentation. The segmentation process predicts binary masks, contours, and distance transforms to enable precise instance segmentation of individual vesicles.
  • Classification:
    • Large Vesicles: A supervised deep learning model (CNN) is used for classifying large vesicles into predefined types based on their morphology.
  • Small Vesicles: An unsupervised approach using variational autoencoders (VAEs) and HDBSCAN clustering is employed to identify and classify small vesicle types without prior manual annotation.
  • Analysis: The pipeline extracts morphological features (volume, radius) and 3D locations (center of mass) for each segmented vesicle. It also generates neuron masks to define regions of interest and analyze vesicle distribution relative to neuronal structures.
  • Visualization: VesiclePy integrates with multiple visualization platforms (Neuroglancer, PyVista, Plotly, and a custom three.js viewer) to allow for inspection of data at various scales, from individual vesicles to entire neuronal networks.

Datasets

The primary dataset used for training and evaluation is serial electron microscopy (SEM) data from Hydra vulgaris. This dataset consists of 1,829 sections, providing a multiterabyte volume. The authors utilized manually annotated ground truth data from 20 complete neurons within this dataset for training and validating their models.

Key Insights and Ideas

  • Bridging Deep Learning and Human Expertise: The pipeline effectively combines the efficiency of deep learning for initial segmentation and classification with the accuracy of human proofreading, creating a robust and reliable analysis tool.
  • Unsupervised Classification for Small Vesicles: The use of VAEs and HDBSCAN for small vesicle classification is a significant contribution, as these vesicles are notoriously difficult to manually annotate and classify. This opens up new avenues for studying their roles.
  • Multi-Platform Visualization: The integration of diverse visualization tools addresses the challenge of presenting complex 3D vEM data, catering to different analytical needs and scales.
  • Neuron Clustering Beyond Morphology: The paper demonstrates that combining vesicle composition with morphological data can lead to more nuanced and accurate clustering of neuron types, suggesting a deeper link between cellular content and neuronal identity.

Key Results

  • VesiclePy successfully processed a multiterabyte serial EM dataset.
  • It automatically annotated 53,851 vesicles from 20 complete neurons.
  • Large vesicles were classified into 5 types (CV, DCV, DCVH, SCV, SDV).
  • The pipeline achieved a validation accuracy of 83.6% for large vesicle classification.
  • The analysis enabled quantitative clustering of neurons into 4 subtypes based on vesicle and morphological data, aligning with qualitative classifications.

Limitations and Future Work

  • Manual Segmentation for Small Vesicles: While large vesicles are handled by deep learning, small vesicles still rely on manual segmentation, which can be a bottleneck for extremely large datasets.
  • Class Imbalance in Large Vesicle Classification: The paper notes that vesicles with mixed morphology (DCVH) showed lower performance in classification, suggesting that class imbalance remains a challenge.
  • Scalability of Visualization Tools: While multiple visualization tools are integrated, some (like PyVista) have performance limitations on full-volume datasets, necessitating further development for extremely large-scale exploration.

Future work could focus on extending the automated segmentation and classification to small vesicles, improving the classification accuracy for challenging vesicle types, and further optimizing visualization tools for massive datasets. The availability of VesiclePy as an open-source tool is expected to facilitate further research in this area.

Summary updated

Abstract

Abstract Vesicles are critical components of neurons that package neurotransmitters and neuropeptides for their release, in order to communicate with other neurons and cells. However, due to their small size, the reconstruction of the full vesicle endowment across an entire neuronal morphology remains challenging. To achieve this, we have used, as a tool to identify and visualize vesicles, Volume Electron Microscopy (vEM), a method that has the nanoscale resolution to detect individual vesicle boundaries, content, and 3D locations. However, the large volume of vEM datasets poses a challenge in the segmentation, classification, and spatial analysis of tens of thousands of vesicles and their target cell in 3D. Here we report the development of VesiclePy, an integrated pipeline for automated segmentation, classification, proofreading, and spatial analysis of vesicles, relative to neuron masks in large-volume electron microscopy data. Our package integrates the efficiency of deep learning and the accuracy of human proofreading and provides a streamlined package in chunked processing and accurate indexing, localization, and visualization of single vesicle resolution in large vEM data. We demonstrate the viability of VesiclePy using high-pressure frozen serial EM data of Hydra vulgaris and quantify the performance of the package using ground truth manual annotations. We show that VesiclePy can process a multiterabyte serial EM dataset, efficiently annotate 53,851 vesicles from 20 complete neurons, and classify vesicles into 5 types. Each vesicle has a unique ID and 3D location for further spatial analysis in relation to neuron or non-neuronal targets nearby. Finally, by combining vesicle data and morphological information of each neuron, we can quantitatively cluster neurons into subtypes. VesiclePy is available at https://github.com/PytorchConnectomics/VesiclePy under an MIT license.
BibTeX
@misc{adhinarta2025vesiclepy,
  author = {Jason Ken Adhinarta and Yutian Fan and Adam Gohain and Michael Che-Hung Lin and Paige Nurkin and Richard Ren and Mark D. Roth and Shulin Zhang and Ayal Yakobe and Rafael Yuste and Donglai Wei},
  title = {{VesiclePy: A Machine Learning Vesicle Analysis Toolbox for Volume Electron Microscopy}},
  year = {2025},
  doi = {10.1101/2025.09.08.674799},
  howpublished = {bioRxiv},
}

Loading related content...

Figure