Crowdsourcing the creation of image segmentation algorithms for connectomics

Ignacio Arganda‐Carreras, Srinivas C. Turaga, Daniel R. Berger, Dan Cireşan, Alessandro Giusti, Luca Maria Gambardella, Jürgen Schmidhuber, Dmitry Laptev, Sarvesh Dwivedi, Joachim M. Buhmann, Ting Liu, Mojtaba Seyedhosseini, Tolga Taşdizen, Lee Kamentsky, Radim Bürget, Václav Uher, Xiao Tan, Changming Sun, Tuan D. Pham, Erhan Bas, Mustafa Gökhan Uzunbaş, Albert Cardona, Johannes Schindelin, H. Sebastian SeungFrontiers in Neuroanatomy 2015 · DOI

Abstract

To stimulate progress in automating the reconstruction of neural circuits, we organized the first international challenge on 2D segmentation of electron microscopic (EM) images of the brain. Participants submitted boundary maps predicted for a test set of images, and were scored based on their agreement with a consensus of human expert annotations. The winning team had no prior experience with EM images, and employed a convolutional network. This "deep learning" approach has since become accepted as a standard for segmentation of EM images. The challenge has continued to accept submissions, and the best so far has resulted from cooperation between two teams. The challenge has probably saturated, as algorithms cannot progress beyond limits set by ambiguities inherent in 2D scoring and the size of the test dataset. Retrospective evaluation of the challenge scoring system reveals that it was not sufficiently robust to variations in the widths of neurite borders. We propose a solution to this problem, which should be useful for a future 3D segmentation challenge.

Introduction

Mapping the "connectome"—the complete wiring diagram of every neuron and synapse—is essential for understanding brain function. Serial section Electron Microscopy (ssTEM) provides the necessary resolution to see these connections, but the resulting data volume is immense. Manual tracing is a significant bottleneck; for instance, reconstructing 1,000 neurons in a mouse retina required roughly 20,000 hours of human labor. To scale connectomics, automated image segmentation must reach human-level precision. This paper describes an open competition designed to attract global computer vision talent to solve this critical hurdle.

Methods and Datasets

The authors organized a two-phase challenge centered on a dataset from the Drosophila (fruit fly) larva ventral nerve cord. The training set consisted of 30 consecutive EM images (512 x 512 pixels) with full human-annotated ground truth, while the test set withheld ground truth for 30 additional images to prevent overfitting.

Overview of the EM image segmentation challenge workflow, from raw micrographs to ground truth and predicted segmentations.

The competition initially ran for four months, culminating at the ISBI 2012 conference. Following this, a "cooperation phase" encouraged teams to share results and code, leading to ensemble approaches. Performance was evaluated using two primary metrics:

  • Rand F-score ($V^{Rand}$): Measures the correct assignment of pixel pairs to the same or different segments.
  • Information Theoretic F-score ($V^{Info}$): Based on mutual information between predicted and ground-truth segmentations.

A significant methodological finding was the need for border thinning. Because different algorithms and human annotators produce varying boundary widths, the authors proposed thinning all predicted boundaries to a single pixel before scoring to ensure a fair evaluation.

Key Insights and Results

The challenge catalyzed a 5x improvement in algorithm accuracy and yielded several transformative insights:

  • The Deep Learning Revolution: The winning team (IDSIA) used a deep convolutional neural network (CNN) that learned features directly from raw pixels. Despite having no prior experience with EM data, they outperformed teams using hand-engineered biological features, marking a permanent shift in the field toward deep learning.
  • The Power of Cooperation: While the initial competition drove a 2.5x improvement, the subsequent cooperation phase—where teams shared probability maps—led to another 2.5x gain.
  • Metric Sensitivity and 2D Saturation: The authors discovered that standard metrics were biased by the thickness of predicted cell walls. Furthermore, the challenge reached a performance "ceiling" due to the inherent ambiguity of 2D slices; some structures require 3D context to be correctly identified.
Comparison of boundary probability maps from different competition entries, highlighting variations in boundary width and local errors.

By the end of the cooperation phase, top algorithms achieved scores statistically indistinguishable from the agreement levels between two human experts. After applying border thinning, the correlation between the Rand and Information Theoretic rankings improved from 0.80 to 0.94, validating the revised scoring method.

Conclusion

While 2D challenges successfully engaged the computer vision community, the "saturation" of results indicates that the future of connectomics lies in 3D segmentation. Moving forward, challenges on larger 3D datasets (such as SNEMI3D) will be necessary to fully capture the complexity of biological neural circuits and overcome the limitations of slice-by-slice analysis.

Summary updated

Abstract

To stimulate progress in automating the reconstruction of neural circuits, we organized the first international challenge on 2D segmentation of electron microscopic (EM) images of the brain. Participants submitted boundary maps predicted for a test set of images, and were scored based on their agreement with a consensus of human expert annotations. The winning team had no prior experience with EM images, and employed a convolutional network. This "deep learning" approach has since become accepted as a standard for segmentation of EM images. The challenge has continued to accept submissions, and the best so far has resulted from cooperation between two teams. The challenge has probably saturated, as algorithms cannot progress beyond limits set by ambiguities inherent in 2D scoring and the size of the test dataset. Retrospective evaluation of the challenge scoring system reveals that it was not sufficiently robust to variations in the widths of neurite borders. We propose a solution to this problem, which should be useful for a future 3D segmentation challenge.
BibTeX
@inproceedings{argandacarreras2015crowdsourcing,
  author = {Ignacio Arganda‐Carreras and Srinivas C. Turaga and Daniel R. Berger and Dan Cireşan and Alessandro Giusti and Luca Maria Gambardella and Jürgen Schmidhuber and Dmitry Laptev and Sarvesh Dwivedi and Joachim M. Buhmann and Ting Liu and Mojtaba Seyedhosseini and Tolga Taşdizen and Lee Kamentsky and Radim Bürget and Václav Uher and Xiao Tan and Changming Sun and Tuan D. Pham and Erhan Bas and Mustafa Gökhan Uzunbaş and Albert Cardona and Johannes Schindelin and H. Sebastian Seung},
  title = {{Crowdsourcing the creation of image segmentation algorithms for connectomics}},
  booktitle = {Frontiers in Neuroanatomy},
  year = {2015},
  doi = {10.3389/fnana.2015.00142},
}
Figure