Clinical studies such as the Alzheimer's Disease Neuroimaging Initiative (ADNI) are small: a few hundred patients, each scanned in 3D. Some patients have every modality, such as structural MRI plus PET, and others have only some. Current methods handle this in one of three ways. They reduce each scan to a few dozen region-averaged numbers, which throws away 3D shape. They feed 2D slices into image models pretrained on natural photos, which lose depth. Or they train 3D networks that overfit at this sample size. On top of that, each missing-modality subgroup becomes its own task, and someone has to hand-tune how much each task's loss counts.
We propose a convex alternative. First, compress every scan with a frozen, pretrained 3D autoencoder, so each patient becomes one whole-volume code. Then learn a patient-similarity kernel, a function scoring how alike two patients are, as a single linear program (an optimization with a linear objective and linear constraints, solvable exactly and quickly). The program minimizes the worst task's loss rather than a hand-weighted sum. Because of this, the task balancing weights come out of the solver as dual variables: optimal, sum to one, and need no tuning. One program covers all missing-modality patterns at once, so its size grows with the number of modalities rather than with the number of modality combinations.
Core Claim
Three gaps in the file meet at this point. Multi-Task Linear Programming Discriminant Analysis, called MLPD below, splits ADNI subjects by which modalities they have. It solves an $\ell_1$-regularized linear program over 93 grey-matter ROI volumes and 93 PET ROI intensities. It raises progressive-MCI sensitivity to 67.8%, against 49–52% for the earlier iMSF method. However, it uses fixed constraint weights and warns that the number of subproblems grows as $2^M$ in the number of modalities $M$.
Compositional Cross-Modality Translation via Whole-Volume Multitask Latent Flow Matching supplies the representation we reuse. Its frozen 3D variational autoencoder (MAISI), pretrained on 55,000 scans, yields whole-volume latents. Whole-volume models built on these latents beat patch-based ones by up to +0.37 SSIM. That paper still trains with plain fixed-weight sums of task losses.
Learning a Kernel for Multi-Task Clustering and Liu et al. supply the optimization trick. Once the eigenvectors of a similarity graph are fixed, learning a kernel reduces to choosing nonnegative eigenvalue weights. That makes it a linear program instead of a semidefinite program, an optimization over positive semidefinite matrices that is much costlier to solve. Both papers assume one shared feature space and a single, unbalanced loss.
Our claim is twofold. First, a single linear program over modality-specific spectral bases, built from whole-volume latents, handles asymmetric missingness without enumerating modality combinations. Second, replacing the weighted sum with a minimax objective turns multi-task loss balancing into a by-product of duality. We predict that at roughly 400 labeled subjects this combination beats both ROI-based linear programs and end-to-end 3D networks with modern loss balancing.
Research Program
Each scan of modality $m$ is encoded by the frozen 3D autoencoder into a latent volume. For each modality we build a nearest-neighbour graph over all subjects that have that modality, including unlabeled ones such as cognitively normal and AD subjects. Using them makes the method transductive: data without labels still shapes the graph. We keep the $K$ smoothest eigenvectors $v_{mk}$ of each graph's Laplacian, with eigenvalues $\lambda_{mk}$, and zero-pad them to the full subject set. The learned kernel is
$$K(\mu) = \sum_{m=1}^{M}\sum_{k=1}^{K} \mu_{mk}\, v_{mk} v_{mk}^{\top}, \qquad \mu_{mk}\ge 0,$$
which is positive semidefinite by construction. A task $t$ is a missingness pattern, for example MRI-only or MRI+PET. It owns the labeled pairs $P_t$ among its subjects and uses only the blocks of the modalities it has. We then solve
$$\min_{\mu,\xi,s}\; s + \gamma \sum_{m,k}\lambda_{mk}\mu_{mk} \quad \text{s.t.}\quad \frac{1}{|P_t|}\sum_{(i,j)\in P_t}\xi_{ij}\le s \;\;\forall t,\quad y_i y_j K_{ij}(\mu)\ge 1-\xi_{ij},\quad \xi\ge 0,\quad \sum\mu_{mk}=1.$$
Here $y_i\in\{\pm1\}$ is the label of subject $i$, $\xi_{ij}$ is a pairwise hinge slack, $s$ bounds the worst task's average slack, and $\gamma$ trades fit against graph smoothness. We also add a linear constraint that $\mu_{mk}$ does not increase with $\lambda_{mk}$.
Every constraint is linear in $\mu$. The dual variables on the per-task constraints are nonnegative and sum to one, so they are exactly the task weights a balancing method would otherwise search for. A per-task SVM is then trained on $K(\mu)$.
The cubic eigendecomposition cost flagged as an open problem for spectral linear programs is negligible here, at under a thousand subjects. We will state that plainly rather than claim scalability we do not need.
We will analyze three things beyond accuracy. The first is the learned dual task weights compared with each pattern's size. The second is how much eigenvalue mass falls on the PET block versus the MRI block. The third is whether high-weight eigenvectors map back to anatomically meaningful latent directions.
Baselines
The naive reference point is a linear SVM on concatenated, flattened autoencoder latents, with mean-imputed PET, trained under equal task weights. Below that sits the trivial majority-class predictor, which we report so that sensitivity gains are not an artefact of class imbalance.
The strongest existing method on this exact question is MLPD. Its paper reports about 67.2% accuracy and 67.8% sensitivity on 393 baseline MCI subjects (167 progressive, 226 stable). We will rerun MLPD on our own splits with its published ROI features, because a matched subject list is not guaranteed. Beating it means higher sensitivity at no lower accuracy, with the gap exceeding the across-fold standard deviation.
A second reference is supervised PLS fusion of MRI, PET and genomics from Genetic, Structural and Functional Imaging Biomarkers, which reports an AUC of 0.72 and 66.14% accuracy from MRI+PET.
The strongest deep competitor is a small 3D network fine-tuned on the same latents and balanced with LDC-MTL, the first-order bilevel loss balancer of Scalable Bilevel Loss Balancing. That method reports a CelebA performance drop of −1.31% against +0.37% for FairGrad. No reported result was found on Lacuna for LDC-MTL on ADNI or any 3D medical data. We also run PSMGD, the periodic multi-gradient method of PSMGD.
For fairness, every method sees identical subjects, folds and autoencoder latents, or the same ROI features where a method needs them. Every method gets the same budget of hyperparameter trials, chosen by nested cross-validation.
Two ablations isolate the key ingredients. The first keeps the same linear program but swaps the minimax objective for a weighted sum whose weights are tuned by cross-validation; this isolates duality-derived balancing. The second fixes $\mu$ to uniform eigenvalue weights; this isolates the spectral kernel learning.
Why not just…?
Why not just fit a linear classifier on the frozen latents with imputed PET? This is the most serious threat. If the frozen autoencoder already makes the classes nearly linearly separable, our machinery adds nothing. The naive baseline above tests this directly. If it matches our method within noise, we will report that whole-volume latents alone carry the gain, and drop the kernel claim.
Why not just fine-tune a 3D network end-to-end with LDC-MTL? Evidence from the file argues against it at this scale. The PLS study found MRI–PET–genomic fusion overfits badly without supervised dimensionality reduction, and Informedia@TRECVID 2013 saw learned fusion weights fail to converge when positives were scarce. Still, neither result tests modern balancers. Our label-subsampling curve, run at 25%, 50% and 100% of labels, will show where, if anywhere, the deep model overtakes the convex one.
Why not just tune fixed task weights by cross-validation? UoloNet reports that loss balances tuned for MRI fail when moved to CT, which suggests tuned weights are brittle. With only two or three tasks, though, a grid search may be adequate. The weighted-sum ablation settles this. We will judge minimax both on mean accuracy and on worst-pattern sensitivity, which is what it is designed to protect.
Why not just slice volumes into a 2D foundation model, as MMGPL does? MMGPL reports 82.3% on ADNI-3CLS and we will compare on that task with one-vs-rest SVMs. However, Your other Left! shows that 2D vision-language models are at chance on medical spatial relations, and MMGPL's authors themselves name the gap between 2D pretraining and 3D volumes as unresolved.
Proposed Experiments
The primary experiment is progressive versus stable MCI on ADNI baseline scans, using 10-fold cross-validation repeated five times. We report accuracy, sensitivity, specificity and AUC, broken down by missingness pattern.
The secondary experiment is ADNI-3CLS, for comparison with MMGPL's reported 82.3%. ABIDE is excluded because its functional-connectivity inputs do not fit a structural autoencoder; no reported number for it was found on Lacuna in any case.
Three decisive checks follow. First, our method must beat rerun MLPD in sensitivity at matched accuracy; if it does not, whole-volume latents do not help over ROIs. Second, the minimax version must improve worst-pattern sensitivity over the cross-validation-tuned weighted sum; if it does not, the balancing claim fails. Third, PET passed through an autoencoder that may never have seen PET must outperform PET ROI features inside the same linear program. The file flags this risk explicitly: frozen 3D autoencoders can fail on modalities outside their training support. If this check fails, we fall back to a hybrid design with latent MRI and ROI PET and report the limitation.
Finally, we sweep the number of modalities to three by adding CSF biomarkers, on the subset that has them. We confirm empirically that linear-program size and solve time grow linearly in $M$, not as $2^M$.
References
- Multi-Task Linear Programming Discriminant Analysis for the Identification of Progressive MCI Individuals
- Compositional Cross-Modality Translation via Whole-Volume Multitask Latent Flow Matching
- Learning a Kernel for Multi-Task Clustering
- Spectral Kernel Learning for Semi-Supervised Classification
- Genetic, Structural and Functional Imaging Biomarkers for Early Detection of Conversion from MCI to AD
- LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Control
- PSMGD: Periodic Stochastic Multi-Gradient Descent for Fast Multi-Objective Optimization
- Informedia@TRECVID 2013
- UoloNet: based on multi-tasking enhanced small target medical segmentation model
- MMGPL: Multimodal Medical Data Analysis with Graph Prompt Learning
- Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images