Interpretable Deep Learning for Pattern Recognition in Brain Differences Between Men and Women

Maxim Kan, Ruslan Aliev, Anna Rudenko, Nikita Drobyshev, Nikita Petrashen, Ekaterina Kondrateva, Maxim Sharaev, Alexander Bernstein, Evgeny Burnaev

Abstract

Deep learning shows high potential for many medical image analysis tasks. Neural networks can work with full-size data without extensive preprocessing and feature generation and, thus, information loss. Recent work has shown that the morphological difference in specific brain regions can be found on MRI with the means of Convolution Neural Networks (CNN). However, interpretation of the existing models is based on a region of interest and can not be extended to voxel-wise image interpretation on a whole image. In the current work, we consider the classification task on a large-scale open-source dataset of young healthy subjects -- an exploration of brain differences between men and women. In this paper, we extend the previous findings in gender differences from diffusion-tensor imaging on T1 brain MRI scans. We provide the voxel-wise 3D CNN interpretation comparing the results of three interpretation methods: Meaningful Perturbations, Grad CAM and Guided Backpropagation, and contribute with the open-source library.

Introduction

In medical imaging, deep learning offers a significant advantage by processing raw, full-size data without the need for manual feature engineering, which often results in information loss. However, the "black box" nature of these models remains a barrier to clinical adoption. Clinicians require transparency to understand why a model reaches a specific diagnosis. Previous interpretation attempts often relied on coarse "Regions of Interest" (ROIs). This study addresses the challenge of voxel-wise interpretation—identifying the importance of every single 3D point in a brain scan—to ensure models focus on actual biological markers rather than noise.

Methods

The researchers utilized the Human Connectome Project (HCP) dataset, comprising T1-weighted structural MRI scans from 1,113 healthy young adults (507 men, 606 women). Scans were downsampled to $58 \times 70 \times 58$ tensors to manage computational constraints.

A lightweight 3D CNN architecture with three hidden layers was employed for classification. To evaluate performance, the CNN was compared against traditional machine learning baselines, including Logistic Regression (LR), XGBoost (XGB), and K-Nearest Neighbors (KNN), trained on 935 pre-extracted morphometry features such as cortical thickness and volume.

To interpret the CNN's decisions, three state-of-the-art methods were compared:

  1. Meaningful Perturbations: Blurring regions to observe drops in model confidence.
  2. Grad-CAM: Using gradients from the final convolutional layer to produce localization maps.
  3. Guided Backpropagation: Visualizing voxels with a positive impact on the final classification score.

Results and Interpretation

The 3D CNN achieved a classification accuracy of 92% (± 0.03), matching the performance of Logistic Regression trained on manually extracted features and slightly outperforming raw-data SVMs (90%).

Table 1: Mean accuracy and standard deviation for baseline machine learning models.

A key finding was the high consensus between the three interpretation methods, achieving DICE overlap scores of over 0.93. This suggests that the model's attention is a robust biological signal. The regions highlighted—including the Orbital Gyrus, Inferior Parietal Lobule, and Corticospinal Tracts—align with established neuroscientific findings on gender-based structural differences.

Figure 1: Voxel-wise interpretation maps comparing (a) Meaningful Perturbations, (b) Guided Backpropagation, and (c) Grad-CAM.

While Grad-CAM proved most practical for clinical workflows due to its speed (under one minute), Meaningful Perturbations provided the most precise, anatomically detailed maps, albeit at a higher computational cost.

Discussion and Conclusion

While the CNN performed admirably, the loss of resolution during downsampling likely prevented it from significantly outperforming traditional methods. Furthermore, results on T1 structural images were slightly lower than those seen in previous Diffusion Tensor Imaging (DTI) studies, suggesting that water diffusion patterns may be more predictive of gender than pure anatomy.

Future work will focus on applying these voxel-wise interpretation tools to pathological cases, such as early-stage Alzheimer’s disease or epilepsy localization, where precise identification of structural anomalies is critical for surgical and clinical outcomes.

Summary updated

Abstract

Deep learning shows high potential for many medical image analysis tasks. Neural networks can work with full-size data without extensive preprocessing and feature generation and, thus, information loss. Recent work has shown that the morphological difference in specific brain regions can be found on MRI with the means of Convolution Neural Networks (CNN). However, interpretation of the existing models is based on a region of interest and can not be extended to voxel-wise image interpretation on a whole image. In the current work, we consider the classification task on a large-scale open-source dataset of young healthy subjects -- an exploration of brain differences between men and women. In this paper, we extend the previous findings in gender differences from diffusion-tensor imaging on T1 brain MRI scans. We provide the voxel-wise 3D CNN interpretation comparing the results of three interpretation methods: Meaningful Perturbations, Grad CAM and Guided Backpropagation, and contribute with the open-source library.
BibTeX
@inproceedings{kan2020interpretable,
  author = {Maxim Kan and Ruslan Aliev and Anna Rudenko and Nikita Drobyshev and Nikita Petrashen and Ekaterina Kondrateva and Maxim Sharaev and Alexander Bernstein and Evgeny Burnaev},
  title = {{Interpretable Deep Learning for Pattern Recognition in Brain Differences Between Men and Women}},
  booktitle = {AIST (Supplement)},
  year = {2020},
  url = {https://openreview.net/forum?id=dUCkHgKcx6g},
}
Figure