NRTR: Neuron Reconstruction with Transformer from 3D Optical Microscopy Images

Yijun Wang, Rui Lang, Rui Li, Junsong ZhangIEEE TMI 2022 · OpenReview

TL;DR

This paper introduces NRTR, a novel Transformer-based model that treats neuron reconstruction as a direct "image-to-set" prediction problem. By eliminating complex rule-based post-processing steps, NRTR enables end-to-end training and achieves state-of-the-art results on challenging 3D microscopy datasets, simplifying the reconstruction pipeline.

What Problem Does This Paper Address?

Understanding the intricate structure of neurons is fundamental to neuroscience research. Traditionally, this involves manually tracing neurons from 3D optical microscopy images, a process that is both time-consuming and prone to human error. While deep learning methods have shown promise, they often rely on a combination of image segmentation networks and complex, hand-designed rule-based algorithms for tracing and connecting neuronal segments. This hybrid approach leads to complicated pipelines, hinders end-to-end training, and can limit performance due to the inherent weaknesses of the rule-based components. The authors aim to address this by developing a simpler, end-to-end trainable model that directly predicts the neuron's structure as a set of points.

Methods

The core innovation of NRTR lies in its formulation of neuron reconstruction as a set-prediction problem, inspired by recent advancements in object detection with Transformers. The NRTR architecture is a hybrid model comprising three main components:

  1. Backbone: A standard ResNet with a Feature Pyramid Network (FPN) is used to extract multi-scale feature maps from the input image blocks. This captures hierarchical information crucial for understanding neuronal structures at different resolutions.
  2. Transformer Encoder-Decoder: The extracted feature maps are tokenized and fed into a Transformer encoder-decoder. Unlike traditional sequence-to-sequence models, NRTR generates a fixed-size set of points simultaneously, rather than sequentially. This is achieved by using a fixed number of "object queries" (or positional embeddings) that learn to attend to relevant image features and predict point properties. The Transformer's self-attention mechanism is key to capturing global dependencies within the image.
  3. Connectivity Construction: While the Transformer predicts the points (coordinates and radius), the paper mentions a "connectivity construction module" that establishes the relationships among these points. This module is described as a rule-based algorithm that embeds the predicted point set into a tree structure, ultimately forming the SWC file format. The authors acknowledge that designing a deep learning module for this step is a future direction.

A crucial aspect of the training process is the set-based loss function. This loss uses the Hungarian algorithm to find the optimal bipartite matching between the predicted points and the ground truth points. This allows for a direct comparison and penalization of mismatches, ensuring that the model learns to predict a diverse and accurate set of points. A "no-object" class is introduced to handle cases where the predicted set size differs from the ground truth.

Datasets

The NRTR model was evaluated on two publicly available datasets:

  • VISOR-40 Dataset: This dataset consists of forty image blocks from mouse brain optical microscopy. It includes both manually annotated and raw image stacks, with a physical resolution of 0.5 x 0.5 x 0.5 µm³ per voxel. The dataset was partitioned into training and testing sets.
  • BigNeuron Dataset: This is a larger-scale dataset sponsored by multiple neuroscience organizations, containing 166 neurons with standard reconstruction results and raw image stacks. Due to its lower resolution, linear interpolation was applied to magnify the image stacks by a factor of eight to improve performance.

Key Insights and Ideas

  • Neuron Reconstruction as Set Prediction: The most significant insight is reframing neuron reconstruction from a segmentation or tracing problem to a direct set-prediction task. This allows for a more unified and end-to-end trainable framework.
  • Transformer for Global Context: Leveraging the Transformer's self-attention mechanism enables the model to capture long-range dependencies and global image context, which is crucial for accurately reconstructing complex neuronal structures.
  • Eliminating Rule-Based Components: By directly predicting points, NRTR successfully removes the need for complex, hand-engineered rule-based algorithms that often plague previous methods, simplifying the pipeline and improving training efficiency.
  • Image-to-Set Paradigm: NRTR is presented as the first image-to-set deep learning model for end-to-end neuron reconstruction, opening up new avenues for tackling similar structured prediction problems in biological imaging.

Key Results

  • State-of-the-Art Performance: NRTR achieved competitive and often superior results compared to existing state-of-the-art methods on both the VISOR-40 and BigNeuron datasets, as measured by Precision, Recall, F-Score, and Jaccard indices.
  • VISOR-40 Dataset: NRTR-ResNet50 surpassed MP-NRGAN by +4.72% on Precision, +8.36% on Recall, +6.33% on F-Score, and +5.52% on Jaccard.
  • BigNeuron Dataset: NRTR-ResNet34 surpassed MP-NRGAN by +9.7% on Recall, +5.34% on F-Score, and +5.84% on Jaccard. NRTR-ResNet50 surpassed MP-NRGAN by +6.01% on Recall, 3.85% on F-Score, and +4.19% on Jaccard.
  • Qualitative Improvements: The paper shows that NRTR generates 3D models with topological structures similar to ground truth, effectively capturing neuronal morphology.

Limitations and Future Work

  • Low-Resolution Image Performance: NRTR's performance is less satisfactory on low-resolution image stacks. While linear interpolation helps, it's not a perfect solution. The authors suggest that the ratio of point radius to input image block size should not be too small for optimal results.
  • Connectivity Construction: The current connectivity construction relies on a rule-based algorithm. The authors propose designing a deep learning module to learn connection information as a key area for future work. This would further enhance the end-to-end nature of the model.
  • Further Improvement: The authors acknowledge that NRTR has "much room for further improvement," particularly in handling challenging low-resolution inputs and fully automating the connectivity aspect.

Summary updated

Abstract

The neuron reconstruction from raw Optical Microscopy (OM) image stacks is the basis of neuroscience. Manual annotation and semi-automatic neuron tracing algorithms are time-consuming and inefficient. Existing deep learning neuron reconstruction methods, although demonstrating exemplary performance, greatly demand complex rule-based components. Therefore, a crucial challenge is designing an end-to-end neuron reconstruction method that makes the overall framework simpler and model training easier. We propose a Neuron Reconstruction Transformer (NRTR) that, discarding the complex rule-based components, views neuron reconstruction as a direct set-prediction problem. To the best of our knowledge, NRTR is the first image-to-set deep learning model for end-to-end neuron reconstruction. In experiments using the BigNeuron and VISoR-40 datasets, NRTR achieves excellent neuron reconstruction results for comprehensive benchmarks and outperforms competitive baselines. Results of extensive experiments indicate that NRTR is effective at showing that neuron reconstruction is viewed as a set-prediction problem, which makes end-to-end model training available.
BibTeX
@article{wang2022nrtr,
  author = {Yijun Wang and Rui Lang and Rui Li and Junsong Zhang},
  title = {{NRTR: Neuron Reconstruction with Transformer from 3D Optical Microscopy Images}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2022},
  url = {https://openreview.net/forum?id=Xv9uNhM4S7},
}
Figure