Large Scale Image Segmentation with Structured Loss Based Deep Learning for Connectome Reconstruction

Jan Funke, Fabian Tschopp, William Grisaitis, Arlo Sheridan, Chandan Singh, Stephan Saalfeld, Srinivas C. Turaga

Abstract

We present a method combining affinity prediction with region agglomeration, which improves significantly upon the state of the art of neuron segmentation from electron microscopy (EM) in accuracy and scalability. Our method consists of a 3D U-Net, trained to predict affinities between voxels, followed by iterative region agglomeration. We train using a structured loss based on Malis, encouraging topologically correct segmentations obtained from affinity thresholding. Our extension consists of two parts: First, we present a quasi-linear method to compute the loss gradient, improving over the original quadratic algorithm. Second, we compute the gradient in two separate passes to avoid spurious gradient contributions in early training stages. Our predictions are accurate enough that simple learning-free percentile-based agglomeration outperforms more involved methods used earlier on inferior predictions. We present results on three diverse EM datasets, achieving relative improvements over previous results of 27, 15, and 250 percent. Our findings suggest that a single method can be applied to both nearly isotropic block-face EM data and anisotropic serial sectioned EM data. The runtime of our method scales linearly with the size of the volume and achieves a throughput of ~2.6 seconds per megavoxel, qualifying our method for the processing of very large datasets.

Introduction

Mapping the "connectome"—the dense web of connections between neurons—is a monumental task in modern neuroscience. 3D electron microscopy (EM) provides the necessary resolution to visualize these connections, but the resulting image volumes are far too large for manual tracing. Automated segmentation faces two primary hurdles: ensuring topological accuracy, where single errors can merge or split distinct neurons, and achieving scalability to handle terabyte-scale datasets.

This paper introduces a highly scalable and accurate framework for reconstructing neural circuits. By combining a 3D U-Net with an optimized "constrained MALIS" structured loss and a linear-time region merging algorithm, we achieve state-of-the-art results across diverse datasets while maintaining the efficiency needed for massive volumes.

Methods

The proposed pipeline consists of three main stages: affinity prediction, structured loss training, and hierarchical agglomeration.

Overview of the proposed segmentation pipeline, including the 3D U-Net architecture, affinity prediction, and agglomeration process.

Affinity Prediction

Instead of labeling voxels as "neuron" or "background," a 3D U-Net is trained to predict "affinities"—the probability that adjacent voxels belong to the same object. This approach is particularly robust for anisotropic data, where resolution is lower in one dimension, as it allows the model to learn connectivity across sections.

Visualization of inter-voxel affinities across multiple sections.

Constrained MALIS Loss

To train the U-Net, we use an improved version of the MALIS (Maximum Labeling Invariant Segmentation) loss. This structured loss specifically penalizes topological errors. We introduce two key optimizations:

  • Quasilinear Computation: We reduced the gradient calculation complexity from $O(n^2)$ to $O(n \log n)$, making it feasible for large volumes.
  • Two-Pass Gradient: Gradients are computed in separate "positive" and "negative" passes to prevent the model from being misled by spurious errors during early training stages.

Hierarchical Agglomeration

To convert affinities into final segments, we use a "seeded watershed" to create initial fragments, followed by iterative merging. By discretizing edge scores into bins and using a bucket priority queue, we achieved $O(n)$ runtime, ensuring the method scales linearly with the size of the dataset.

Datasets and Results

The method was evaluated on three diverse EM benchmarks:

  • CREMI (ssTEM, Drosophila): Highly anisotropic with significant artifacts.
  • FIB-25 (FIBSEM, Drosophila): Nearly isotropic.
  • SEGEM (SBEM, mouse cortex): Isotropic mammalian brain data.

The method achieved relative improvements over previous state-of-the-art results by 27% on FIB-25, 15% on CREMI, and 250% on SEGEM. The system achieves a throughput of approximately 2.6 seconds per megavoxel. The linear-time agglomeration scheme proved significantly faster than traditional $O(n \log n)$ priority-queue-based merging.

Key Insights

  • Affinities over Labels: Predicting connectivity between voxels is more robust than predicting voxel classes, especially when boundaries are thin or resolution is low in one dimension.
  • Topological Awareness: Training a network to care about connectivity (via MALIS) rather than just pixel values leads to much more usable biological reconstructions.
  • Simplicity Scales: If initial affinity predictions are accurate enough, simple "learning-free" merging strategies (like percentile-based agglomeration) can outperform more complex, computationally expensive methods.
  • Unified Approach: A single architectural framework can handle both isotropic and anisotropic data.

Conclusion and Future Work

While the algorithm is linear in time and memory, parallelizing the hierarchical agglomeration step remains a challenge for "infinite" scale processing. Future work will also involve direct comparisons with newer models like Google’s Flood-Filling Networks (FFNs) and refining the stitching methods used for extremely large volumes.

Summary updated

Abstract

We present a method combining affinity prediction with region agglomeration, which improves significantly upon the state of the art of neuron segmentation from electron microscopy (EM) in accuracy and scalability. Our method consists of a 3D U-Net, trained to predict affinities between voxels, followed by iterative region agglomeration. We train using a structured loss based on Malis, encouraging topologically correct segmentations obtained from affinity thresholding. Our extension consists of two parts: First, we present a quasi-linear method to compute the loss gradient, improving over the original quadratic algorithm. Second, we compute the gradient in two separate passes to avoid spurious gradient contributions in early training stages. Our predictions are accurate enough that simple learning-free percentile-based agglomeration outperforms more involved methods used earlier on inferior predictions. We present results on three diverse EM datasets, achieving relative improvements over previous results of 27, 15, and 250 percent. Our findings suggest that a single method can be applied to both nearly isotropic block-face EM data and anisotropic serial sectioned EM data. The runtime of our method scales linearly with the size of the volume and achieves a throughput of $\sim$∼ 2.6 seconds per megavoxel, qualifying our method for the processing of very large datasets.
BibTeX
@article{funke2018large,
  author = {Jan Funke and Fabian Tschopp and William Grisaitis and Arlo Sheridan and Chandan Singh and Stephan Saalfeld and Srinivas C. Turaga},
  title = {{Large Scale Image Segmentation with Structured Loss Based Deep Learning for Connectome Reconstruction}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2018},
  doi = {10.1109/tpami.2018.2835450},
}
Figure