Accurate and fast segmentation of filaments and membranes in micrographs and tomograms with TARDIS

Robert Kiewisz, Gunar Fabig, William F. Conway, Jake Johnston, V.a. Kostyuchenko, Cyril Bařinka, Oliver B. Clarke, Magdalena M. Magaj, Hossein Yazdkhasti, Francesca Vallese, Shee‐Mei Lok, Stefanie Redemann, Thomas Müller‐Reichert, Tristan BeplerbioRxiv 2024 · DOI

TL;DR

Researchers have developed TARDIS, a novel deep learning framework that automates the segmentation of filaments and membranes in electron microscopy images and tomograms. This tool dramatically reduces annotation time from months to minutes, enabling large-scale quantitative biophysical analysis that was previously infeasible.

What Problem Does This Paper Address?

Analyzing the structure and organization of biomolecules using electron microscopy (EM) and electron tomography (ET) is crucial for understanding cellular processes. However, a major bottleneck is the manual segmentation (annotation) of these structures, which is incredibly time-consuming, often taking months per dataset. This manual effort is not only slow but also prone to variability. Existing automated tools are often limited in their applicability, accuracy, or require extensive manual tuning, leaving a significant gap for fast, accurate, and generalizable segmentation of complex cellular structures like filaments and membranes.

Methods

TARDIS is built upon a modular deep learning framework that combines state-of-the-art semantic and instance segmentation capabilities.

  • FNet Architecture: For semantic segmentation, TARDIS uses a custom-designed FNet, a CNN architecture with a dual-decoder structure. This design allows for capturing features at multiple scales and generating sharper probability maps compared to traditional UNet models.
  • DIST Model: For instance segmentation, TARDIS employs a novel geometric transformer architecture called DIST (Dimensionless Instance Segmentation Transformer). DIST operates on point clouds derived from semantic masks and uses an SO(n) equivariant representation to capture geometric relationships in a translation- and rotation-invariant manner, making it robust to variations in object size and orientation.
  • Pipeline: The overall workflow involves image pre-processing, semantic segmentation by FNet, conversion of semantic masks to point clouds via skeletonization and voxel down-sampling, and finally, instance segmentation by DIST.

Datasets

The authors assembled several datasets for training and evaluation:

  • Microtubule Datasets: A comprehensive collection of 93 tomograms (cryo and room temperature) with resolutions from 1.16 to 3.64 nm, encompassing various imaging techniques and specimen preservation methods. They also used TIRF microscopy images of microtubules.
  • Membrane Datasets: A diverse dataset of 53 tomograms (0.55 to 2.40 nm resolution) and micrographs, curated by experts and supplemented with data from EMPIAR and CZI Cryo-ET portals.
  • Actin Datasets: 3 tomograms with resolutions of 1.38 nm and 1.81 nm.
  • Point Cloud Synthetic Dataset: A dataset combining actual and simulated point clouds for training the DIST model.
  • Benchmark Datasets: The paper introduces a new benchmark dataset of hand-curated microtubules and membranes, comprising 91 tomograms and 112 TIRF images for microtubules, and 120 micrographs and 53 tomograms for membranes. This is claimed to be the largest manually labeled dataset of its kind.

Key Insights and Ideas

  • Transformer for Instance Segmentation: The application of a geometric transformer (DIST) for instance segmentation on point clouds is a novel approach that effectively handles complex structures and potential overlaps.
  • Modularity and Generalizability: TARDIS's modular design allows for easy integration of new segmentation networks, making it adaptable to various biomolecules and imaging modalities without extensive retraining.
  • Bridging the Gap: TARDIS directly addresses the critical bottleneck of manual annotation, enabling large-scale quantitative analysis that was previously computationally prohibitive.
  • Performance Across Modalities: The framework demonstrates strong performance across diverse datasets, including cryo-ET, plastic ET, cryo-EM micrographs, and TIRF microscopy.

Key Results

  • Speed: TARDIS reduces annotation time from months to minutes, processing over 13,000 tomograms in one month.
  • Accuracy: TARDIS improves annotation accuracy by 42% for microtubules and 55% for membranes compared to existing tools. It achieves an mCov score of 0.72 for microtubules compared to Amira's 0.42. For membranes, TARDIS achieves an mCov score of 0.73, more than doubling MemBrain-seg V2's performance (0.33).
  • Efficiency: TARDIS offers a 2x speedup for membrane segmentation and 5x for microtubule segmentation compared to other algorithms.
  • Versatility: The tool successfully segments various structures like microtubules, membranes, actin filaments, and Tobacco Mosaic Virus (TMV) without retraining. It also shows promise for segmenting other point cloud data, such as LiDAR scans.
  • Large-Scale Analysis: TARDIS enabled the analysis of variability in kinetochore microtubule (KMT) numbers in U2OS cells and detailed quantification of DENV-liposome fusion intermediates.

Limitations and Future Work

  • Limitations: While TARDIS demonstrates broad applicability, the paper notes that some existing tools require significant manual effort or are limited to specific structures. The performance of TARDIS on new, unseen datasets might still benefit from fine-tuning or fallback to manual annotation in some cases. The paper also mentions that variability in KMT numbers and lengths could reflect cell-specific differences or poor expression of phosphomutants, suggesting that expanding analysis to more cells is essential for confirmation.
  • Future Work: The authors envision TARDIS evolving into a comprehensive framework capable of annotating all cellular structures with the continuous addition of new segmentation networks. They also highlight the potential for TARDIS to be extended to new biomolecules and applications. The open-source nature of TARDIS and its Napari plugin are intended to foster an open science environment and accelerate future innovations.

Summary updated

Abstract

Abstract Segmentation of macromolecular structures is the primary bottleneck for studying biomolecules and their organization with electron microscopy in 2D/3D – requiring months of manual effort. Transformer-based Rapid Dimensionless Instance Segmentation (TARDIS) is a deep learning framework that automatically and accurately annotates membranes and filaments. Pre-trained TARDIS models can segment electron tomography (ET) reconstructions from both 3D and 2D electron micrographs of cryo and plastic-embedded samples. Furthermore, by implementing a novel geometric transformer architecture, TARDIS is the only method to provide accurate instance segmentations of these structures. Reducing the annotation time for ET data from months to minutes, we demonstrate segmentation of membranes and filaments in over 13,000 tomograms in the CZII Data Portal. TARDIS thus enables quantitative biophysical analysis at scale for the first time. We show this in application to kinetochore-microtubule attachment and viral-membrane interactions. TARDIS can be extended to new biomolecules and applications and open-source at https://github.com/SMLC-NYSBC/TARDIS .
BibTeX
@misc{kiewisz2024accurate,
  author = {Robert Kiewisz and Gunar Fabig and William F. Conway and Jake Johnston and V.A. Kostyuchenko and Cyril Bařinka and Oliver B. Clarke and Magdalena M. Magaj and Hossein Yazdkhasti and Francesca Vallese and Shee‐Mei Lok and Stefanie Redemann and Thomas Müller‐Reichert and Tristan Bepler and Tristan Bepler},
  title = {{Accurate and fast segmentation of filaments and membranes in micrographs and tomograms with TARDIS}},
  year = {2024},
  doi = {10.1101/2024.12.19.629196},
  howpublished = {bioRxiv},
}
Figure