Abstract
Deep learning, particularly 3D Convolutional Neural Networks (CNNs), offers significant potential for medical image analysis by processing raw data without extensive manual feature engineering. However, these models often operate as "black boxes," making interpretation difficult. This paper explores the classification of gender differences in brain MRI scans using a large-scale dataset of young healthy subjects. We compare three voxel-wise interpretation methods—Meaningful Perturbations, Grad-CAM, and Guided Backpropagation—to identify stable, neuroanatomically relevant patterns. Our results demonstrate that 3D CNNs achieve high classification accuracy and identify consistent structural differences that align with established neuroscience literature.
Introduction
While CNNs have shown success in identifying morphological differences in brain MRIs, most existing models lack granular, voxel-wise interpretation across the whole image. Understanding the specific features a model uses for classification is critical in clinical settings. This study addresses this gap by applying state-of-the-art interpretability tools to a 3D CNN trained on gender classification using the Human Connectome Project (HCP) dataset, which includes T1-weighted structural MRI scans from 1,113 healthy young adults.
Methodology
We implemented a lightweight 3D CNN architecture and compared its performance against traditional machine learning baselines, including Support Vector Machines (SVM), XGBoost, K-Nearest Neighbors (KNN), and Logistic Regression. The latter models were trained on "morphometry" features (e.g., cortical volume and surface area) extracted via FreeSurfer.
To interpret the CNN's decision-making process, we employed three distinct techniques:
- Meaningful Perturbations: Identifying critical regions by measuring the drop in model confidence when specific areas are blurred.
- Guided Backpropagation: Visualizing gradients to highlight voxels that positively influence classification.
- Grad-CAM: Using final-layer gradients to produce coarse localization maps of important regions.
Key Results
The 3D CNN achieved a mean classification accuracy of 0.92 ± 0.03, matching the performance of Logistic Regression trained on hand-crafted features and slightly outperforming the full-size image SVM (0.90). The baseline performance for other traditional models is summarized below:
A critical finding was the high stability across interpretation methods. The DICE scores (measuring overlap) between the attention maps generated by the three techniques were above 0.93, indicating that the model identifies consistent biological signals regardless of the visualization method used.
Discussion and Neuroanatomical Validation
The regions identified as most discriminative by the CNN—including the Orbital Gyrus, Inferior Parietal Lobule, and Corticospinal tracts—align closely with existing literature on gender-related structural brain differences. Interestingly, the attention maps revealed patterns resembling the frontoparietal resting-state network, a hub for cognitive control. This suggests that structural variations in these functional hubs are primary drivers for the model's classification.
While the CNN performed well, the study noted that significant data downsampling (to $58 \times 70 \times 58$ tensors) was required to manage GPU memory. This compression likely limited the CNN's ability to outperform traditional models by losing fine-grained structural details.
Conclusion
This work demonstrates that 3D CNNs can reliably identify neuroanatomically meaningful patterns in brain MRI data. By providing a voxel-wise interpretation framework, we move closer to making deep learning models transparent and clinically useful. The authors have released the code as an open-source library to facilitate the application of these methods to other 3D medical imaging tasks, such as biomarker identification for Alzheimer’s disease or autism.
Summary updated