TL;DR
This paper introduces NeurINO, a novel method for 3D neuron segmentation that leverages powerful 2D self-supervised visual priors from DINOv3. By "inflating" 2D convolutional filters into 3D and incorporating a topology-aware loss, NeurINO achieves state-of-the-art performance in reconstructing complex neuronal structures with significantly less data than traditional methods.
What Problem Does This Paper Address?
Accurately reconstructing the intricate 3D morphology of neurons is crucial for understanding brain connectivity and function. However, this task is incredibly challenging due to the fine, branching, and often low-contrast nature of neuronal arbors, compounded by the scarcity of high-quality, annotated 3D neuroimaging data. Existing deep learning approaches often struggle to preserve the topological fidelity of these structures, leading to fragmented or incomplete reconstructions, even when achieving high voxel-wise accuracy. The paper aims to bridge this gap by effectively transferring knowledge from large-scale 2D visual foundation models to the 3D domain, enabling more data-efficient and morphologically accurate neuron segmentation.
Methods
The core of NeurINO lies in its innovative approach to adapting 2D self-supervised visual priors to 3D neuroimaging.
- DINOv3 Initialization: The 3D segmentation encoder is initialized by "inflating" the pre-trained 2D weights from DINOv3. This strategy injects robust, generalizable visual features learned from natural images into the 3D model, providing a strong starting point for understanding neuronal structures.
- Inflation-Based Adaptation: A key technique is the spatial inflation of 2D convolutional kernels into 3D. This is achieved through two schemes: "center inflation" (placing the 2D kernel at the center of the 3D kernel) and "average inflation" (uniformly replicating the 2D kernel along the depth). Center inflation is found to be more effective, preserving spatial semantics and alignment.
- Topology-Aware Skeleton Loss (TASL): To explicitly enforce structural fidelity, a novel TASL is introduced. This loss function penalizes discrepancies in predicted neuron skeletons at three levels: node (geometric deviation), edge (connectivity consistency), and path (long-range continuity). This goes beyond simple voxel overlap to ensure the reconstructed arbors are topologically sound.
- Encoder-Decoder Architecture: NeurINO uses a DINOv3-inflated encoder coupled with a symmetric MedNeXt-style decoder. Deep supervision is applied across multiple decoder outputs, jointly supervised by Dice loss, cross-entropy loss, and the proposed TASL.
Datasets
The method was evaluated on four publicly available volumetric neuroimaging datasets:
- Drosophila: Two volumes from BigNeuron.
- Mouse: Two volumes from BigNeuron.
- NeuroFly: A dataset constructed from VISOR and fMOST imaging.
- CWMBS: A dataset of mouse brain volumes.
The paper does not introduce new datasets but utilizes existing benchmarks to demonstrate the effectiveness of its approach.
Key Insights and Ideas
- 2D Priors are Powerful for 3D: The success of NeurINO demonstrates that large-scale 2D self-supervised models like DINOv3 can be effectively adapted to significantly improve performance on 3D biomedical tasks, even with limited 3D data.
- Topology Matters More Than Voxel Accuracy: The paper highlights a crucial distinction: high voxel-wise segmentation accuracy doesn't always translate to accurate neuron reconstruction. The TASL is essential for capturing the complex, continuous structure of neurons.
- Inflation Strategy is Key: The "inflation" of 2D kernels is not just a simple expansion; the "center inflation" strategy is critical for preserving the spatial relationships and semantic information learned by DINOv3, leading to better feature learning in 3D.
- Smaller Encoders Can Still Excel: While a larger DINOv3 backbone (Small) generally yields slightly better segmentation, the Tiny variant, when combined with the inflation and TASL, achieves superior reconstruction performance, suggesting that efficient adaptation is more important than sheer model size for this task.
Key Results
- Segmentation: NeurINO consistently outperforms state-of-the-art (SoTA) segmentation baselines (nnUNet, MedNeXt, TransUNet, SegMamba) on F1 scores and HD95 across all four datasets. For example, NeurINOS achieves an F1 score of 50.19% on Drosophila, outperforming others.
- Reconstruction: NeurINO models achieve significant improvements in neuron tracing metrics (ESA, DSA, PDS) compared to other methods. NeurINOT, in particular, shows superior reconstruction performance, with average gains of 2.9% in ESA, 2.8% in DSA, and 3.8% in PDS.
- Efficiency: NeurINO achieves competitive or lower computational costs (parameters and FLOPs) compared to other 3D segmentation models like nnUNet and MedNeXt.
- Ablation Studies: Center inflation is shown to be superior to average inflation. The full TASL (node, edge, and path components) is crucial for optimal reconstruction, even if removing components slightly improves segmentation metrics. Full finetuning of the DINOv3 encoder is essential.
Limitations and Future Work
- Approximation to Volumetric Understanding: The authors acknowledge that NeurINO, while effective, is still an approximation of true volumetric understanding.
- Future Directions: They suggest exploring genuinely 3D foundation models pretrained on large-scale neuroimaging datasets. They also emphasize the need for broader open access to large 3D volumes to support the development of universal 3D neuronal models.
Summary updated