Abstract
Deep Learning (DL) and specifically CNN models have become a de facto method for a wide range of vision tasks, outperforming traditional machine learning (ML) methods. Consequently, they drew a lot of attention in the neuroimaging field in particular for phenotype prediction or computer-aided diagnosis. However, most of the current studies often deal with small single-site cohorts, along with a specific pre-processing pipeline and custom CNN architectures, which make them difficult to compare to. We propose an extensive benchmark of recent state-of-the-art (SOTA) 3D CNN, evaluating also the benefits of data augmentation and deep ensemble learning, on both Voxel-Based Morphometry (VBM) pre-processing and quasi-raw images. Experiments were conducted on a large multi-site 3D brain anatomical MRI data-set comprising N=10k scans on 3 challenging tasks: age prediction, sex classification, and schizophrenia diagnosis. We found that all models provide significantly better predictions with VBM images than quasi-raw data. This finding evolved as the training set approaches 10k samples where quasi-raw data almost reach the performance of VBM. Moreover, we showed that linear models perform comparably with SOTA CNN on VBM data. We also demonstrated that DenseNet and tiny-DenseNet, a lighter version that we proposed, provide a good compromise in terms of performance in all data regime. Therefore, we suggest to employ them as the architectures by default. Critically, we also showed that current CNN are still very biased towards the acquisition site, even when trained with N=10k multi-site images. In this context, VBM pre-processing provides an efficient way to limit this site effect. Surprisingly, we did not find any clear benefit from data augmentation techniques. Finally, we proved that deep ensemble learning is well suited to re-calibrate big CNN models without sacrificing performance.
Introduction
While deep learning has revolutionized computer vision through standardized benchmarks like ImageNet, the neuroimaging field has struggled with a lack of consistency. Most studies rely on small, single-site cohorts and custom architectures, making it difficult to determine if performance gains are robust or merely overfitted to specific niches. This paper establishes a rigorous benchmark using a massive aggregated dataset to evaluate 3D CNN architectures, the impact of pre-processing, and the efficacy of data augmentation and ensemble learning.
Methods and Datasets
The study utilizes the BHB-10K dataset, comprising 10,420 T1-weighted MRI scans from 13 public sources. Models were evaluated on three tasks: age prediction (regression), sex classification, and schizophrenia diagnosis.
The researchers compared several architectures adapted for 3D: VGG, ResNet, ResNeXt, and DenseNet, alongside a proposed tiny-DenseNet. A core focus was the comparison between two pre-processing pipelines:
- Voxel-Based Morphometry (VBM): An extensive pipeline involving non-linear registration and tissue segmentation (Gray Matter maps).
- Quasi-Raw: A minimalist approach using only linear registration and bias field correction.
Key Results and Insights
Pre-processing vs. Scaling
In small data regimes ($N < 1,000$), VBM significantly outperforms quasi-raw data. Interestingly, with VBM data, simple linear models (e.g., Ridge Regression) perform almost as well as complex CNNs. However, as the training set size approaches 10,000, the performance gap between quasi-raw and VBM data begins to close, suggesting that CNNs can eventually learn to extract relevant features from noisier, less processed data.
The Challenge of Site Bias and Augmentation
Even with 10,000 training samples from dozens of sites, CNNs remain heavily biased toward the specific scanner or site they were trained on. While VBM pre-processing helps mitigate this "site signature" better than raw data, it does not eliminate it. Furthermore, common MRI-specific data augmentations—such as simulating motion, ghosting, or spike artifacts—provided little to no benefit and occasionally degraded performance.
Model Calibration and Ensembles
Large CNNs are often overconfident in their predictions. The authors demonstrated that Deep Ensembles (training multiple identical models with different initializations) are highly effective at re-calibrating these models, making their probability outputs more reliable for clinical decision-making without sacrificing predictive performance.
Conclusion
DenseNet and tiny-DenseNet emerged as the most efficient "default" choices for 3D brain MRI tasks. While scaling to 10k samples improves the utility of raw data, persistent site bias remains a major hurdle for clinical deployment. Future work must focus on domain adaptation techniques to decouple biological signatures from scanner-specific noise.
Summary updated