Deep networks trained on 3D brain scans from a few hundred patients often lose to linear classifiers. They have far more parameters than there are subjects, and they latch onto scanner quirks instead of disease. We propose a different way to spend those few hundred scans. Before asking a network to diagnose anything, we teach it to read anatomy. Every scan comes with a registration-derived map of how much each small region of the brain has shrunk or grown, and that map supplies a large number of correct training targets at no labeling cost. We arrange these targets into a ladder that runs from coarse to fine. The network first predicts whole-lobe volume changes, then volumes of smaller structures, then age, then noisy cognitive test scores, and only last the clinical diagnosis. The earlier tasks stay in the training mix as anchors, so the network does not overwrite what it learned first.
The input representation is tensor-based morphometry. The method warps each subject's MRI onto a common brain template and records, at every voxel, the Jacobian determinant of the warp. This is a number above 1 where the subject's tissue is larger than the template and below 1 where it has shrunk. These maps are already strong disease markers. The bet here is that their spatial structure also defines a natural curriculum, meaning an easy-to-hard ordering of training tasks: averages over large regions are easy, low-noise targets, and finer or more clinical targets are progressively harder. If the bet holds, a 3D network trained this way should beat both a plain multi-task network and linear baselines in exactly the small-cohort regime where deep models currently fail.
Core Claim
In the high-dimensional, small-sample regime of Jacobian maps, the order in which a shared 3D network meets its tasks matters as much as which tasks it meets. Training proceeds in four stages. It begins with label-free regional morphometric targets at coarse spatial scale and moves to finer scales. Next comes demographic regression (age), then cognitive score regression, and finally diagnosis or conversion. Each new stage keeps the earlier tasks as anchors. We claim this ladder will produce diagnostic accuracy that varies less across random seeds and training-set sizes, and that is higher at small training-set sizes, than the same network and the same tasks trained all at once.
Concretely, at stage $s$ the network minimizes
$$\mathcal{L}_s = \sum_{k \le s} \lambda_{k,s}\, \mathcal{L}_k(\theta),$$
where $\mathcal{L}_k$ is the loss of task $k$ on the ladder, $\theta$ the shared network weights, and $\lambda_{k,s}$ the weight on task $k$ during stage $s$. Weights are zero for tasks not yet introduced and decay toward a small floor for older tasks rather than toward zero. The decay to a floor, rather than removal, is the "mixed curriculum" ingredient. It is taken from How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes. That paper trained a small Transformer on synthetic linear, quadratic, and cubic function classes. A model trained only on quadratics converged in 0% of runs. A model given linear-then-mixed exposure converged in 60% of runs with one ninth of the data. The same paper found that a purely sequential curriculum forgot the earlier tasks. Its setting was noiseless 1D synthetic functions. Ours is a noisy 3D anatomical field with a few hundred subjects, and whether its finding transfers there is an open question.
What is new is the source and shape of the ladder. The easy tasks come from the input modality itself: regional means of the log-Jacobian over atlas parcels at multiple scales. They therefore exist for every scanned subject, including those without cognitive scores. Their difficulty is set by spatial scale, not by a human ranking of example difficulty. This sidesteps the known problem in curriculum learning that teacher-defined difficulty need not match the network's own loss landscape. The coarse regional targets are, by construction, nearly linear functions of the input.
Relation to Prior Work
Three papers motivate the gap by showing that Jacobian maps carry signal but that existing learners use them with a single objective or a fixed pipeline.
Enabling uncertainty estimation in neural networks through weight perturbation for improved Alzheimer's disease classification trains a 3D residual CNN on 96×96×96 Jacobian maps from 376 ADNI (Alzheimer's Disease Neuroimaging Initiative) subjects as a single-task Alzheimer's-versus-control classifier. It reaches 86% accuracy, rising to 95% when the 25% most uncertain cases are rejected through final-layer weight perturbation. We keep a comparable backbone and input. What we change is the training signal, from one binary label to a staged task ladder.
Predicting Brain Amyloid Using Multivariate Morphometry Statistics, Sparse Coding, and Correntropy handles the same small-sample problem in a different way. It compresses hippocampal surface tensor morphometry with patch-wise sparse coding and then applies a random forest, validated externally on OASIS (Open Access Series of Imaging Studies). That is a disconnected, hand-staged pipeline. Ours is a single learned representation.
Multi-Method Analysis of MRI Images in Early Diagnostics of Alzheimer's Disease constrains the proposal most. It shows that tensor-based morphometry reaches 90% sensitivity and 84% specificity for Alzheimer's versus controls on ADNI, and that linear discriminant analysis matches or beats support vector machines. Any deep method must therefore clear a strong linear bar, and we treat that bar as a primary baseline rather than a footnote.
On the multi-task side, HiMAL shows that routing cognitive sub-score forecasts into a conversion classifier lifts AUROC (area under the ROC curve) from 0.875 for single-task to 0.902 for parallel multi-task and 0.923 for hierarchical multi-task. It does this on tabular volumes of 90 regions, with cognitive scores also used as inputs. HiMAL supplies our evidence that cognitive auxiliary tasks help conversion prediction. We differ in three ways: we work from voxel-level deformation fields, we add label-free anatomical rungs below the cognitive tasks, and we order the tasks in time rather than only in architecture.
RotoGrad addresses conflict between tasks inside a joint optimization. It reweights tasks by learning speed and applies per-task rotations of the shared features. It is the strongest optimizer-level alternative to our temporal ordering, and we run it as a baseline.
Training Medical Image Analysis Systems like Radiologists combines curriculum ideas with small-sample medical imaging. Our research file describes it only partially, so we claim no detailed contrast beyond this: it orders training examples, whereas we order tasks drawn from deformation geometry.
Research Program
We will compute Jacobian determinant maps for ADNI T1 scans. Each scan is registered to a study template with a single fixed registration pipeline, and the log-determinant is stored on a 96×96×96 grid to match the uncertainty paper. The same pipeline will be applied to OASIS for external testing.
The ladder has four rungs. The first rung is regression of mean log-Jacobian over coarse parcels (lobes) and then fine parcels (subcortical nuclei, hippocampal subfields where the atlas allows). These targets are computed from the input and are available for every scan. The second rung is chronological age regression. Age is available for all subjects and depends on anatomy only nonlinearly. The third rung is regression of cognitive scores such as MMSE and ADAS-Cog, which are noisy and partially missing; missing labels are masked. The fourth rung is classification: Alzheimer's versus control, and progressive versus stable mild cognitive impairment (MCI). Stage transitions are triggered when validation loss on the newest rung plateaus, not by a fixed epoch count. This removes most of the pacing hyperparameters that make self-paced learning sensitive to tuning, as reported for Self-Paced Learning for Latent Variable Models. The anchor floor $\lambda_{\min}$ is the one added hyperparameter, and it is tuned on validation folds.
The analysis focuses on behavior at small sample sizes. We subsample the labeled training pool to 50, 100, and 200 subjects and to the full set. The rung-1 targets remain computable for all available scans in every condition, since they need no labels. We will also check that the network reads anatomy rather than scanner identity. We do this by testing whether a site classifier trained on the frozen shared features weakens after rung 1 compared with a network trained only on diagnosis, and by measuring the drop from ADNI to OASIS.
Baselines
The naive reference is a linear model on regional means. We compute the same parcel-wise mean log-Jacobian features that serve as rung-1 targets and feed them to linear discriminant analysis with shrinkage. This is the regime in which Multi-Method Analysis of MRI Images in Early Diagnostics of Alzheimer's Disease found linear methods to match or beat nonlinear ones, and where its tensor-based morphometry reached 90% sensitivity and 84% specificity for Alzheimer's versus controls. It is the real "do nothing clever" competitor. It uses identical subject splits, its shrinkage is tuned by the same inner cross-validation, and the proposal beats it only if balanced accuracy is higher at every training-set size with confidence intervals that do not overlap.
The strongest published deep competitor on the same input is the single-task 3D residual network of Enabling uncertainty estimation in neural networks through weight perturbation, which reports 86% accuracy for Alzheimer's versus control on 376 ADNI subjects without rejection and 95% at 25% rejection. We reimplement it on our splits rather than comparing against the printed number, since splits differ. We give it the same epoch budget and hyperparameter search as our method, and we also apply its final-layer perturbation to our model so that both are compared at matched rejection rates. Beating it means higher accuracy both without rejection and at 25% rejection.
For progressive-versus-stable MCI from Jacobian maps, we found no reported result on Lacuna for a deep model. The Multi-Method paper's numbers for this split are not given in our research file. HiMAL's 0.923 AUROC uses cognitive scores as inputs and a different prediction window, so it is cited as context, not as a matched target.
The multi-task competitors use the same backbone and the full task set trained jointly from the start. One version uses uniform weights. The other uses RotoGrad, following RotoGrad: Gradient Homogenization in Multi-Task Learning. Both get the same tuning budget, measured in total configurations tried, as the ladder.
The ablations isolate the key ingredient, which is ordering with retained anchors. A sequential version removes the anchors. An anti-curriculum version reverses the rung order. A shuffled version randomizes the order of rungs. A version without rung 1 keeps the curriculum but drops the label-free morphometric targets. The claim survives only if the full ladder beats joint training, the anchor-free sequential version, and the reversed order.
Why not just…?
Why not just use shrinkage LDA on regional Jacobian means? This is a serious threat, and the proposal is designed so that it could lose. Our rung-1 targets are exactly these features, so a network that learns only them adds nothing. The proposal earns its place only if voxel-level structure carries diagnostic signal beyond parcel averages and the ladder lets a deep network reach that signal without overfitting. If shrinkage LDA ties at 50 and 100 subjects, we will report that the benefit appears, if at all, only at larger cohorts.
Why not just train all tasks jointly with a gradient-balancing method? RotoGrad resolves conflicts between tasks within a step, but every task still pulls on a randomly initialized network from the first step. The function-class study found that exposure order, not task mix alone, decided whether hard tasks converged. The comparison between joint RotoGrad training and the ladder, at matched compute, tests whether that finding transfers. If RotoGrad ties, ordering is unnecessary and we will say so.
Why not just pretrain on the anatomical targets and then fine-tune on diagnosis? This is the anchor-free sequential ablation. The function-class paper and the forgetting results cited in our multi-task notes, such as Wang et al., predict that fine-tuning on roughly 100 labels will erode the anatomical features. Our representation-probe experiment measures that erosion directly by tracking rung-1 error during the final stage.
Why not just use self-paced learning over subjects? Ordering examples by loss decides which subjects the network sees early, not what the network learns to compute. With only a few hundred subjects, hiding the hard ones throws away scarce data. We include a self-paced variant on the diagnosis task alone as a secondary comparison.
Proposed Experiments and Decisive Checks
The primary experiment uses ADNI with subject-level five-fold cross-validation, repeated over five seeds, at the four training-set sizes. The endpoints are balanced accuracy and AUROC for Alzheimer's versus control and for progressive versus stable MCI. External validation trains on ADNI and tests on OASIS, following the protocol of the amyloid sparse-coding paper, which reported 0.79 on ADNI and 0.81 on OASIS for its own amyloid task.
Three checks would be decisive. First, the full ladder must beat joint training, with and without RotoGrad, on progressive-versus-stable MCI at 100 labeled subjects, and its spread across seeds must be smaller. Second, it must at least match shrinkage LDA on regional means at every training-set size and beat it at some size. Third, the reversed-order and anchor-free ablations must perform worse than the full ladder; otherwise the gain comes from having more tasks, not from their order.
A secondary check is whether the ladder reduces how much site information the shared features carry and narrows the ADNI-to-OASIS gap. That would support the claim that anatomy-first training steers a small-sample network away from scanner shortcuts. If the first check fails while the auxiliary tasks still help, the outcome is a negative result for curriculum learning and a positive one for label-free morphometric multi-task learning, and we will report it that way.
References
- How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes
- Enabling uncertainty estimation in neural networks through weight perturbation for improved Alzheimer's disease classification
- Predicting Brain Amyloid Using Multivariate Morphometry Statistics, Sparse Coding, and Correntropy: Validation in 1,101 Individuals From the ADNI and OASIS Databases
- Multi-Method Analysis of MRI Images in Early Diagnostics of Alzheimer's Disease
- HiMAL: A Multimodal Hierarchical Multi-task Auxiliary Learning framework for predicting and explaining Alzheimer disease progression
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Training Medical Image Analysis Systems like Radiologists
- Self-Paced Learning for Latent Variable Models
- Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling