Multi-level feature fusion network for neuronal morphology classification

Chunli Sun, Feng ZhaoFrontiers in Neuroscience 2024 · DOI

TL;DR

This paper introduces a novel "Multi-level Feature Fusion Network" (MLFFN) that significantly improves neuronal morphology classification by intelligently fusing hand-crafted morphometric features with deep learning-derived features. By employing a unique "Multi-Level Fusion Module" (MLFM), the network effectively captures complementary information between these feature types, leading to state-of-the-art classification accuracy.

What Problem Does This Paper Address?

Classifying neurons based on their intricate shapes (morphology) is crucial for understanding brain function, but it's a challenging task. Neurons come in a vast array of types, reconstructions can be imperfect, and defining subtle differences is difficult. Researchers have explored two main avenues for describing neuron shapes:

  1. Hand-crafted morphometrics: These are statistical measurements derived from neuron structures, like branch lengths, bifurcation counts, or soma size. While useful for capturing local details, they can struggle with highly complex dendrites.
  2. Deep features: These are learned representations from neural networks, particularly effective at capturing the overall structure of neurons from images. However, converting 3D neuron data into 2D images for these networks can lead to a loss of information.

Existing methods often rely on using only one type of feature or simply concatenating them. This approach fails to fully exploit the complementary nature of hand-crafted and deep features, limiting classification performance. The authors aim to bridge this gap by developing a method that effectively fuses these diverse representations at multiple levels.

Methods

The core innovation of this paper is the Multi-Level Feature Fusion Network (MLFFN), which integrates information from two distinct branches:

  • Hand-crafted Morphometrics Branch: A shallow Fully Connected Network (FCN) processes pre-computed statistical features (like those from the L-Measure toolbox).
  • Deep Features Branch: A Convolutional Neural Network (CNN), specifically ResNet-50, extracts features from 2D projections of the 3D neuron data.

The key to the MLFFN is the Multi-Level Fusion Module (MLFM), which is strategically embedded within the feature extraction blocks of both branches. The MLFM comprises two sub-modules:

  • Feature Enhancement Module (FEM): This module uses channel attention to highlight the most discriminative features within each individual branch (either the FCN or CNN output).
  • Feature Interaction Module (FIM): This module employs a cross-attention mechanism to facilitate communication and information exchange between the FCN and CNN branches. It allows the network to learn how features from one branch complement or interact with features from the other.

This multi-level interaction ensures that the network not only enhances individual feature representations but also learns the synergistic relationships between them, leading to a more robust and distinctive neuronal descriptor. Finally, the enhanced features from both branches are fused (using element-wise summation in their experiments) and fed into a classifier.

Datasets

The authors evaluated their method on three publicly available datasets:

  • NeuroMorpho-10: This dataset contains 5,000 neurons from 10 different types, sourced from various species and brain regions.
  • NeuroMorpho-12: This dataset, used for generalization testing, includes 12 types of cells.
  • NeuroMorpho-17: Another dataset for generalization testing, this one features 17 different cell types and a larger number of cells (47,460).

Key Insights and Ideas

  • Complementarity is Key: The paper strongly emphasizes that hand-crafted morphometrics and deep features are not redundant but rather complementary. One captures local details, while the other captures global structure, and their fusion is more powerful than either alone.
  • Fusion Beyond Concatenation: Simply concatenating features is insufficient. The MLFM's ability to facilitate interaction and information exchange between feature representations at multiple levels is crucial for achieving superior performance.
  • Attention for Enhanced Representation: Both channel attention (FEM) and cross-attention (FIM) play vital roles in refining feature representations and enabling effective inter-branch communication.
  • Multi-Level Integration: Embedding fusion modules at multiple stages of feature extraction, rather than just at the end, allows for a more nuanced and integrated understanding of neuronal morphology.

Key Results

The proposed MLFFN achieved state-of-the-art results:

  • NeuroMorpho-10: Achieved an accuracy of 95.18%, significantly outperforming methods that used only CNNs (85.44%) or FCNs (82.40%) individually.
  • Generalization: The model demonstrated strong generalization capabilities on the NeuroMorpho-12 (92.00% accuracy) and NeuroMorpho-17 (90.10% accuracy) datasets, even with fewer training samples than some comparative methods.
  • Ablation Studies: The results from ablation studies confirmed the importance of both the FEM and FIM modules, with the full MLFFN achieving the highest accuracy. Different fusion strategies (concatenation, average, element-wise summation) were also evaluated, with element-wise summation yielding the best results.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Model Complexity: The MLFM, while effective, increases model complexity and computational cost. Future work will focus on reducing this complexity, potentially by decreasing feature dimensionality within the MLFM or exploring linearly complex modules for FIM.
  • Data Sparsity and Noise: The 2D projected images used for the CNN branch can be sparse and contain significant background, which may lead to overfitting. The authors suggest exploring data cleaning techniques, regularization strategies, and tailored data augmentation methods (e.g., branch deformation, random masks) to improve robustness.
  • Early Stopping: Observing fluctuations in loss and accuracy during training suggests that implementing early stopping could be beneficial to prevent overfitting.
  • Model Simplification: Pruning layers or reducing feature dimensionality are proposed as ways to enhance model capacity and efficiency.

Summary updated

Abstract

Neuronal morphology can be represented using various feature representations, such as hand-crafted morphometrics and deep features. These features are complementary to each other, contributing to improving performance. However, existing classification methods only utilize a single feature representation or simply concatenate different features without fully considering their complementarity. Therefore, their performance is limited and can be further improved. In this paper, we propose a multi-level feature fusion network that fully utilizes diverse feature representations and their complementarity to effectively describe neuronal morphology and improve performance. Specifically, we devise a Multi-Level Fusion Module (MLFM) and incorporate it into each feature extraction block. It can facilitate the interaction between different features and achieve effective feature fusion at multiple levels. The MLFM comprises a channel attention-based Feature Enhancement Module (FEM) and a cross-attention-based Feature Interaction Module (FIM). The FEM is used to enhance robust morphological feature presentations, while the FIM mines and propagates complementary information across different feature presentations. In this way, our feature fusion network ultimately yields a more distinctive neuronal morphology descriptor that can effectively characterize neurons than any singular morphological representation. Experimental results show that our method effectively depicts neuronal morphology and correctly classifies 10-type neurons on the NeuronMorpho-10 dataset with an accuracy of 95.18%, outperforming other approaches. Moreover, our method performs well on the NeuronMorpho-12 and NeuronMorpho-17 datasets and possesses good generalization.
BibTeX
@inproceedings{sun2024multi,
  author = {Chunli Sun and Feng Zhao},
  title = {{Multi-level feature fusion network for neuronal morphology classification}},
  booktitle = {Frontiers in Neuroscience},
  year = {2024},
  doi = {10.3389/fnins.2024.1465642},
}
Figure