Stability of AI-Enabled Diagnosis of Parkinson’s Disease: A Study Targeting Substantia Nigra in Quantitative Susceptibility Mapping Imaging

Bin Xiao, Naying He, Qian Wang, Feng Shi, Zenghui Cheng, E. Mark Haacke, Fuhua Yan, Dinggang ShenFrontiers in Neuroscience 2021 · DOI

Abstract

Purpose: Parkinson's disease (PD) diagnosis algorithms based on quantitative susceptibility mapping (QSM) and image algorithms rely on substantia nigra (SN) labeling. However, the difference between SN labels from different experts (or segmentation algorithms) will have a negative impact on downstream diagnostic tasks, such as the decrease of the accuracy of the algorithm or different diagnostic results for the same sample. In this article, we quantify the accuracy of the algorithm on different label sets and then improve the convolutional neural network (CNN) model to obtain a high-precision and highly robust diagnosis algorithm.

Methods: The logistic regression model and CNN model were first compared for classification between PD patients and healthy controls (HC), given different sets of SN labeling. Then, based on the CNN model with better performance, we further proposed a novel "gated pooling" operation and integrated it with deep learning to attain a joint framework for image segmentation and classification.

Results: The experimental results show that, with different sets of SN labeling that mimic different experts, the CNN model can maintain a stable classification accuracy at around 86.4%, while the conventional logistic regression model yields a large fluctuation ranging from 78.9 to 67.9%. Furthermore, the "gated pooling" operation, after being integrated for joint image segmentation and classification, can improve the diagnosis accuracy to 86.9% consistently, which is statistically better than the baseline.

Conclusion: The CNN model, compared with the conventional logistic regression model using radiomics features, has better stability in PD diagnosis. Furthermore, the joint end-to-end CNN model is shown to be suitable for PD diagnosis from the perspectives of accuracy, stability, and convenience in actual use.

Introduction

Quantitative Susceptibility Mapping (QSM) has emerged as a powerful MRI technique for measuring iron deposition in the brain, a critical biomarker for Parkinson’s Disease (PD). Specifically, clinicians focus on iron levels within the substantia nigra (SN). However, a significant "human-in-the-loop" bottleneck exists: to analyze the SN, an expert must first manually label or segment its boundaries.

Different experts—or even different automated algorithms—rarely agree perfectly on these boundaries. This study addresses the resulting labeling instability. If an input mask changes slightly because of different expert interpretations, traditional machine learning models often produce inconsistent diagnostic results. This research sought to build a model that remains stable and accurate regardless of the labeling source.

Methodology and Framework

The researchers evaluated two primary diagnostic approaches across varying levels of labeling precision to determine their robustness:

  1. Radiomics + Logistic Regression (LR): A traditional pipeline where hand-engineered features (shape, texture, intensity) are extracted from the labeled SN region and processed by a linear classifier.
  2. PDNet (CNN): A deep learning approach that learns hierarchical features directly from the QSM image.
Overview of the experimental workflow comparing manual and automatic labeling pipelines for Parkinson's disease diagnosis.

To address the instability problem, the authors introduced a novel "Gated Pooling" operation. Standard CNNs can be overly sensitive to the "sharp edges" created by binary masks. Gated pooling utilizes a probabilistic map rather than a hard binary mask to weight voxel importance, effectively "softening" the boundaries and reducing sensitivity to minor labeling errors.

Furthermore, the team integrated their segmentation network, SN2, with the classification network (PDNet) into a unified end-to-end framework. This allows the system to process a raw QSM image and output a diagnosis without requiring manual boundary drawing.

The architecture of the SN2 network used for automated substantia nigra labeling.

Experimental Setup and Datasets

The study utilized a dataset of 140 subjects from Ruijin Hospital, Shanghai, consisting of 87 PD patients and 53 age- and sex-matched healthy controls (HC). All subjects underwent 3T MRI to produce QSM maps.

To test stability, the authors used a registration algorithm (SyN) to "warp" ground-truth manual labels. This created a series of synthetic label sets with decreasing Dice coefficients (from 1.000 down to 0.872) to mimic varying levels of expert disagreement.

Key Insights

  • The Fragility of Radiomics: Traditional radiomics features proved highly sensitive to mask variations. When the SN label deviated slightly, the Logistic Regression model’s performance dropped significantly, indicating that hand-engineered features lack the robustness required for clinical deployment.
  • CNN Robustness: The CNN-based PDNet was significantly more stable. By learning hierarchical spatial features, it can effectively ignore minor boundary noise that confuses simpler statistical models.
  • Solving the "Edge Effect": Masking an image creates artificial high-contrast edges that can distract standard CNNs. Gated pooling filters this noise, focusing the network on the internal tissue properties of the substantia nigra.
  • End-to-End Efficiency: The joint framework eliminates the "propagation of error" common in multi-stage pipelines, providing a more reliable and convenient diagnostic tool.

Results and Discussion

The experimental results highlighted a clear divide in stability:

  • Stability: The CNN maintained a stable accuracy of approximately 86.4% across different label sets. Conversely, the Logistic Regression model fluctuated wildly, with accuracy dropping from 78.9% to 67.9% as label precision decreased.
  • Peak Performance: The proposed joint framework (segmentation + gated pooling classification) achieved a consistent 86.9% accuracy.
  • Consistency: Using a "Classification Consistency Index" (CCI), the authors demonstrated that gated pooling significantly increased the likelihood of the model providing the same diagnosis for a patient despite labeling variations.

Conclusion

Compared to conventional radiomics-based models, the CNN model demonstrates superior stability and accuracy in PD diagnosis. The proposed joint end-to-end CNN framework offers a robust, accurate, and convenient solution for clinical use, mitigating the impact of expert labeling variability in QSM imaging.

Summary updated

Abstract

<b>Purpose:</b> Parkinson's disease (PD) diagnosis algorithms based on quantitative susceptibility mapping (QSM) and image algorithms rely on substantia nigra (SN) labeling. However, the difference between SN labels from different experts (or segmentation algorithms) will have a negative impact on downstream diagnostic tasks, such as the decrease of the accuracy of the algorithm or different diagnostic results for the same sample. In this article, we quantify the accuracy of the algorithm on different label sets and then improve the convolutional neural network (CNN) model to obtain a high-precision and highly robust diagnosis algorithm. <b>Methods:</b> The logistic regression model and CNN model were first compared for classification between PD patients and healthy controls (HC), given different sets of SN labeling. Then, based on the CNN model with better performance, we further proposed a novel "gated pooling" operation and integrated it with deep learning to attain a joint framework for image segmentation and classification. <b>Results:</b> The experimental results show that, with different sets of SN labeling that mimic different experts, the CNN model can maintain a stable classification accuracy at around 86.4%, while the conventional logistic regression model yields a large fluctuation ranging from 78.9 to 67.9%. Furthermore, the "gated pooling" operation, after being integrated for joint image segmentation and classification, can improve the diagnosis accuracy to 86.9% consistently, which is statistically better than the baseline. <b>Conclusion:</b> The CNN model, compared with the conventional logistic regression model using radiomics features, has better stability in PD diagnosis. Furthermore, the joint end-to-end CNN model is shown to be suitable for PD diagnosis from the perspectives of accuracy, stability, and convenience in actual use.
BibTeX
@inproceedings{xiao2021stability,
  author = {Bin Xiao and Naying He and Qian Wang and Feng Shi and Zenghui Cheng and E. Mark Haacke and Fuhua Yan and Dinggang Shen},
  title = {{Stability of AI-Enabled Diagnosis of Parkinson’s Disease: A Study Targeting Substantia Nigra in Quantitative Susceptibility Mapping Imaging}},
  booktitle = {Frontiers in Neuroscience},
  year = {2021},
  doi = {10.3389/fnins.2021.760975},
}
Figure