Deep networks for brain MRI face an awkward mismatch. A structural scan has around a million voxels, a typical clinical cohort has a few hundred patients, and the disease signal we care about most, the earliest stage of cognitive decline, is subtle and spread across several brain structures. We propose a simple change to where a 3D convolutional network collects its evidence. Instead of averaging features over the whole volume, the network averages them separately inside each region of a standard anatomical atlas (a labeled map of hippocampus, ventricles, cortical areas and so on). The classifier then sees one short feature vector per brain region rather than a whole-brain summary or a million voxels. The encoder still reads the full 3D volume with its native spatial continuity, but the decision can only be built from anatomically indexed evidence, and the number of quantities the classifier must fit is set by the atlas rather than the scan.
The point is not only to build a better classifier. It is to answer a question the field keeps talking around: once a model is forced to describe the brain region by region, is there diagnostic signal in the internal 3D pattern of each region (local shape, boundary thinning, texture) beyond the region's volume or mean thickness? Graph methods assume the answer is no and reduce each region to a scalar. End-to-end 3D CNNs assume the answer is yes but never test it. Early mild cognitive impairment (EMCI), where whole-brain 3D CNNs sit near chance, is the setting where the answer matters most.
Core Claim
The claim is that region-wise pooling of a 3D encoder's feature maps improves small-sample discrimination of early neurodegenerative stages (EMCI vs. normal controls, and three-way AD/MCI/NC) over two reference points. The first is whole-brain global average pooling on the same encoder. The second is linear models on the same atlas's regional summaries. We further claim that the improvement comes from within-region spatial pattern and not from the pooling acting as a crude volume estimator. That second part is what makes the proposal falsifiable. If a model given only the atlas label map, with no intensities, matches the full model, then the claim is wrong and regional volumes are all there is.
Let $f(v) \in \mathbb{R}^d$ be the encoder's feature vector at spatial location $v$ in the final feature map, and let $m_r(v) \in [0,1]$ be the soft membership of location $v$ in atlas region $r$, downsampled to feature-map resolution. The pooled descriptor for region $r$ is
$$z_r = \frac{\sum_v m_r(v)\, f(v)}{\sum_v m_r(v)}, \qquad r = 1, \dots, R,$$
and the classifier is a linear map on the concatenation $[z_1, \dots, z_R]$. It fits $R \cdot d$ weights, for example 100 regions × 16 channels, instead of the roughly $10^6$ weights a voxelwise linear model fits. The soft masks come from a probabilistic atlas or from averaging several registrations. They exist because hard binary masks break at ambiguous tissue borders, a limitation the file flags through Probabilistic Radiomics, and because they limit the damage when a single segmentation fails.
Relation to Prior Work
Two nearby lines of work supply the mechanism, and neither was tested in the regime that matters here. Computed tomography–based COVID–19 triage through a deep neural network using mask–weighted global average pooling weights a ResNet-50's feature maps by a single lung mask and gains 0.9 points of sensitivity and 2 points of specificity over plain pooling, but it uses one foreground mask, thoracic CT, and 2,809 scans. Lesion-Centric Latent Phenotypes weights 2D ultrasound feature maps by a lesion mask and reaches 0.982 AUC with a linear probe. Our design differs in three ways. It uses many regions rather than one foreground, so the readout keeps multi-region structure instead of just suppressing background. It works in native 3D. And it targets the $p \gg N$ neuroimaging regime these papers never touched.
The motivating evidence comes from volumetric neuroimaging. Residual and Plain Convolutional Neural Networks for 3D Brain MRI Classification trains whole-brain 3D CNNs on 231 ADNI subjects without any segmentation. It reaches 0.88 AUC for AD vs. NC but only 0.57 (VoxCNN) and 0.58 (ResNet) for EMCI vs. NC. End-To-End Alzheimer's Disease Diagnosis and Biomarker Identification gets 94.1% for AD vs. NC but 61.1% for three classes. Both argue explicitly against anatomical priors. Their early-stage numbers are the gap we target, and their argument that ROI selection throws away distributed context is exactly why we pool over all atlas regions rather than cropping a few.
Two further lines constrain the design. Benchmarking CNN on 3D Anatomical Brain MRI and Exploring the potential of representation and transfer learning for anatomical neuroimaging show three things: 3D CNNs trained from scratch do not beat ridge regression on VBM gray-matter maps below about 1,000 subjects; they absorb scanner-site signatures; and pretraining on large healthy cohorts is what lets deep models pull ahead. We therefore never compare a from-scratch deep model against a tuned linear model and call it a win. Every deep arm uses the same pretrained encoder. We also measure site leakage directly.
The graph approaches reduce each region to scalars. GTAD uses 160 regions with multimodal summaries and reports 96.3% preclinical accuracy. Boosting brain connectome classification accuracy in Alzheimer's disease using higher-order singular value decomposition works on connectomes. These papers represent the hypothesis our key ablation tests, namely that the scalar summary is sufficient. Anatomical Foundation Models for Brain MRIs (AnatCL) is the closest relative across all three directions. It uses FreeSurfer-derived regional metrics, but only as distances inside a contrastive pretraining loss. Its forward pass pools globally. We reuse its pretrained 3D ResNet-18 and change only the readout, which makes AnatCL the natural backbone rather than a competitor.
Research Program
We build one encoder and vary only how its features reach the classifier. The encoder is a 3D ResNet-18 initialized from AnatCL weights, pretrained on OpenBHB healthy scans (code at the AnatCL repository referenced in that paper). A second copy is randomly initialized to show whether pretraining is necessary. Inputs are minimally processed T1 volumes registered to a standard template. Atlas masks come from the same FreeSurfer or CAT12 pipeline AnatCL uses. Soft masks are formed by Gaussian smoothing of the label map at feature-map resolution, following the smoothing used in the mask-weighted pooling CT work.
The analysis has three parts. First, a learning-curve study subsamples the training set at 50, 100, 200 and all available subjects per class. The dimensionality argument predicts that region pooling should help most at the smallest $N$ and converge toward whole-brain pooling as $N$ grows, mirroring the quasi-raw-vs-VBM convergence Dufumier et al. report near 10,000 subjects. Second, an information-content study removes kinds of within-region information one at a time (described under decisive checks). Third, a site-leakage study trains a logistic-regression probe on frozen pooled features to predict acquisition site on OpenBHB, and compares region-pooled with globally pooled features. This tests the file's open question of whether anatomical filtering removes scanner signatures or merely moves them around.
Baselines
The naive reference point is a majority-class predictor together with the published whole-brain 3D CNN numbers. For EMCI vs. NC, Residual and Plain Convolutional Neural Networks for 3D Brain MRI Classification reports AUC 0.57 (VoxCNN) and 0.58 (ResNet). For three-class AD/MCI/NC, End-To-End Alzheimer's Disease Diagnosis and Biomarker Identification reports 61.1% accuracy. Those numbers come from different splits, so we also re-implement VoxCNN on our splits and treat the published figures only as orientation.
The strongest structural-only competitor we expect is a regularized linear model on the same atlas's regional summaries: volume, mean intensity, and, for cortical regions, thickness and surface area. We fit elastic net and linear SVM. This is the "regions as scalars" hypothesis in its most efficient form, and Dufumier et al. show linear models are hard to beat at our sample sizes. A second linear competitor is ridge regression on whole-brain VBM gray-matter maps, the configuration Dufumier et al. found matches 3D CNNs below 1,000 subjects. The file gives no reported number for this configuration on ADNI early-stage tasks (no reported result found on Lacuna for VBM ridge on EMCI vs. NC), so we run it ourselves.
The strongest deep competitor is the AnatCL-pretrained encoder with ordinary global average pooling and a linear head, both frozen and fine-tuned. No reported result found on Lacuna for AnatCL's numeric ADNI performance, so again we run it. MVSC reports 97.16% binary AUC on ADNI CN vs. AD, and we include it on that task where code permits. That figure is for a much easier contrast, however, and we do not claim to compete with it there. GTAD's 96.3% uses PET and diffusion data and is not a structural-only comparison. We cite it as context, not as a target.
Four ablations isolate the key ingredient. The first uses a single whole-brain mask, which is the direct 3D analogue of the COVID mask-weighted pooling. The second uses hard binary masks instead of soft ones. The third uses a random parcellation with the same number and size distribution of regions, which tests whether anatomical boundaries matter or just any partition that reduces dimension. The fourth pools only over a hippocampus-plus-ventricle subset.
Fairness is enforced through shared conditions. All methods use identical subject-level splits, and no subject's scans appear in both training and test. Evaluation uses five repetitions of stratified five-fold cross-validation plus an external test on OASIS-3, where MVSC reports 86.41% binary AUC as a reference. Every method gets the same nested hyperparameter search of 30 configurations, and all deep arms get the same GPU budget and number of epochs.
Beating the strongest baseline means an EMCI-vs-NC AUC gain over the regional-summary elastic net whose bootstrap 95% interval excludes zero, at two or more training-set sizes, with the external OASIS-3 AUC not degrading. It also means beating the random-parcellation ablation, since otherwise the gain is dimensionality reduction rather than anatomy.
Why not just…?
Why not just use ridge regression on VBM gray-matter maps? This is the most credible threat, because Dufumier et al. show it matches 3D CNNs at our sample sizes. VBM already embeds a tissue segmentation, and it is voxelwise, so it can in principle capture within-region pattern. Its weakness is statistical: it fits about $10^6$ weights from a few hundred subjects, and nothing ties them to anatomy. The learning curves settle this directly. If VBM ridge equals region pooling at every $N$, we report that the architectural change adds nothing and that pretraining alone is not enough to break linear parity on early-stage tasks.
Why not just feed regional volumes and thicknesses to a linear model or graph network, as GTAD does? This is the null hypothesis of the whole proposal, so it is a primary baseline and not an afterthought. The mask-only and texture-flattened conditions described below test directly whether anything beyond such summaries is being used. If the scalar model wins, the honest conclusion is that within-region 3D pattern carries no usable early-stage signal at these sample sizes. That would be a useful negative result for a field that routinely argues both ways.
Why not just crop a box around the hippocampus? Hard cropping works for localized targets such as joints (Interpretable Rheumatoid Arthritis Scoring via Anatomy-aware Multiple Instance Learning) and organs in CT reports (Resolution Meets Reduction). Early Alzheimer's changes, however, are distributed, which is the point both Korolev et al. and Esmaeilzadeh et al. make. The medial-temporal-subset ablation is effectively this baseline run fairly. If it matches full-atlas pooling, the extra regions are unnecessary and we say so.
Why not just pretrain better and keep global pooling? Pretraining is what Dufumier et al. found lets deep models beat linear ones. That is why the AnatCL-pretrained, globally pooled encoder is the main deep baseline, with pretraining held fixed so that only the readout changes. If global pooling on that encoder matches region pooling, the readout is irrelevant once representations are good.
Proposed Experiments and Decisive Checks
The primary benchmark is ADNI structural T1. The tasks are EMCI vs. NC, LMCI vs. NC, AD vs. EMCI (all with published VoxCNN references of 0.57, 0.67 and 0.66 AUC) and the three-way AD/MCI/NC split used by Esmaeilzadeh et al. External validity is checked on OASIS-3, and site leakage on OpenBHB. Metrics are AUC with bootstrap intervals, balanced accuracy for the three-class task, and site-probe balanced accuracy. We also produce region-level attribution maps (the magnitude of each $z_r$'s contribution to the linear head) and check their stability across folds.
The decisive check is a set of three input conditions run through the identical region-pooled model. The first is the real scan. The second is a "flattened" scan in which every voxel is replaced by its region's mean intensity. This keeps region shape and volume but erases internal texture. The third is the atlas label map alone, which keeps only shape. If the real scan beats the flattened scan on EMCI vs. NC with a non-overlapping interval, within-region pattern carries signal that scalar-summary methods discard. If the flattened scan beats the label map, intensity summaries add value beyond shape. If all three tie, and tie with the elastic net, regional morphometry is the whole story at this sample size.
The second check is the random-parcellation ablation, which shows whether anatomical boundaries matter beyond dimension reduction. The third is the site probe. Region pooling is only a partial answer to site bias if its site-probe accuracy is not meaningfully below that of global pooling. Together these three checks turn a small architectural change into a direct test of what a 3D network can learn about early neurodegeneration from a few hundred brains.
References
- Probabilistic Radiomics: Ambiguous Diagnosis with Controllable Shape Analysis
- Computed tomography–based COVID–19 triage through a deep neural network using mask–weighted global average pooling
- Lesion-Centric Latent Phenotypes from Segmentation Encoders for Breast Ultrasound Interpretability
- Residual and plain convolutional neural networks for 3D brain MRI classification
- End-To-End Alzheimer's Disease Diagnosis and Biomarker Identification
- Benchmarking CNN on 3D Anatomical Brain MRI: Architectures, Data Augmentation and Deep Ensemble Learning
- Exploring the potential of representation and transfer learning for anatomical neuroimaging: Application to psychiatry
- Multi-modal Graph Neural Network with Transformer-Guided Adaptive Diffusion for Preclinical Alzheimer Classification
- Boosting brain connectome classification accuracy in Alzheimer's disease using higher-order singular value decomposition
- Anatomical foundation models for brain MRIs
- Multimodal Visual Surrogate Compression for Alzheimer's Disease Classification
- Interpretable Rheumatoid Arthritis Scoring via Anatomy-Aware Multiple Instance Learning
- Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generation