Clinical prediction often combines several kinds of data about one patient, such as a dermoscopy photo, a clinical photo and a few metadata fields, or a pathology slide and a gene-expression profile. Models that let these sources interact inside the network, rather than averaging separate predictions at the end, tend to do better on cohorts of several hundred to a thousand patients. Clinical cohorts are often smaller than that. At 50 to 300 patients, the interaction machinery memorizes, and practitioners fall back on frozen features fed to a random forest or an SVM. This note proposes a simple way to give an expressive fusion model far more training examples without inventing any pixels or numbers. Take two patients with the same diagnosis and swap one modality between them. With $n_c$ patients in class $c$ and $M$ modalities, this yields up to $n_c^M$ label-consistent combinations, and every modality's features in each combination come from a real patient.
The swap has a known cost. It keeps everything each modality says about the label but destroys any dependence between modalities that goes beyond the label, which is exactly what an interaction-capable model could exploit. We therefore treat the fraction of swapped pairs as a dial between two models: an additive, late-fusion-like model (all swapped) and a fully interacting one (all real). We study where that dial should sit as cohort size changes. We also propose matching swap partners by risk so that within-class severity correlations survive, and we use test-time swapping as a diagnostic of how much any fusion model actually relies on cross-modal interaction.
Core Claim
In small multimodal cohorts, most of the overfitting in deep fusion comes from estimating cross-modal interactions from too few real pairings. Training a fusion head on a mixture of real and label-conditional recombined pairs should beat both pure late fusion and unregularized deep fusion below roughly a few hundred patients. The best mixture fraction should fall as the cohort grows. Risk-matched recombination, which swaps a modality only between same-class patients whose out-of-fold unimodal risk scores are close, should keep most of the benefit while losing less of the genuine cross-modal signal than uniform within-class swapping. The new elements are the recombination augmentation, its explicit sample-size-dependent tradeoff, and the swap-based diagnostic of interaction use. None of these appears in the prior work we found.
Relation to Prior Work
Three groups of papers motivate the problem. The first shows that interaction pays off at moderate sample sizes. TFormer fuses clinical and dermoscopic images stage by stage and then adds metadata, yet still reports overfitting on Derm7pt's sub-500 training set. SkinM2Former mixes the three modalities across depths and beats late-fusion models. HEALNet routes slides and omics through a small shared latent array and reports up to 4.5% over existing fusion strategies on TCGA cohorts of several hundred patients or more. CA-MLIF combines cross-attention with low-rank fusion. All four train the fusion parameters on real pairs only and do not test below about 100 patients.
The second group shows that at small sample sizes, simple pipelines on frozen features win. Frozen text embeddings fed to a random forest beat fine-tuning at $N=72$ (Alhuzali et al.). LASSO-selected deep and radiomics features fed to an SVM beat an end-to-end 3D ResNet on 106 aneurysm cases (Liu et al.). This group defines the bar we must clear.
Three further papers constrain the design. Nelson et al. regularize a stacking combiner by sampling synthetic meta-features from class-conditional Gaussians in log-probability space. That is the closest existing idea, since it also creates class-conditional synthetic multimodal inputs. It differs from ours in three ways. It is parametric, so it needs density estimates that the authors report break down in a minority-class "danger zone" of about 150 patients. It works only on base-model outputs, not on representations a fusion head can attend over. It helps tree combiners but not linear ones. Recombination needs no density estimate and supplies combinatorially many samples even for tiny classes. MixUp-MIL shows that interpolating features across slides hurts because it creates unrealistic mixtures. We read this as a warning against blending within a modality. Swapping whole modalities never blends, so each modality block remains a real observation. Jha et al. show that late fusion fails mainly because modality encoders learn at conflicting speeds, which they fix with per-modality learning rates. We borrow this as a training detail.
Research Program
All modality encoders are frozen pretrained networks, which removes encoder overfitting as a confound and isolates the fusion head. Above these encoders sits a fusion head that can model interactions: a small cross-attention block over modality tokens, comparable in size to a HEALNet-style latent array. Each training batch contains a fraction $\rho$ of recombined examples. A recombined example is built by fixing a class $c$, drawing a patient from that class for each modality independently, and labelling the assembled tuple $c$.
Uniform within-class drawing assumes that modalities are independent given the label. That assumption is false whenever, for instance, a larger lesion on dermoscopy goes with a larger lesion in the clinical photo. Risk-matched recombination relaxes it. For each modality we compute out-of-fold unimodal risk scores, and we swap modality $m$ only among same-class patients whose risk scores from the other modalities fall in the same quantile bin. This preserves the coarse severity correlation while still multiplying the number of examples. Recombination with dropped modalities is also natural, which gives a direct route to robustness against missing modalities.
The analysis instrument is the pairing gap. For any trained fusion model, it is the difference in held-out performance between real test pairs and within-class swapped test pairs. A gap near zero means the model behaves additively. A large gap means it relies on patient-specific cross-modal dependence, which can be either genuine or memorized. We expect three things. The pairing gap of the best model should grow with $N$. Large gaps at small $N$ should predict poor generalization. The best $\rho$ should be predictable from a cohort's sample size and from the pairing gap measured on its largest available subset.
Baselines
The naive reference point is the best single modality, a linear probe on the strongest frozen encoder, alongside plain concatenation of frozen features with logistic regression. Any fusion claim must beat these at every cohort size.
The strongest practical competitor at small $N$ is frozen features with a classical learner. We run two versions: XGBoost or a random forest on concatenated frozen features, and the LASSO-plus-SVM pipeline of Liu et al.. On Cyst-X we also run the cUPMI-regularized stacking ensemble exactly as described by Nelson et al.. That paper reports a binary AUC of 0.839 and a quadratic weighted kappa (QWK) of 0.595 for its random-forest stack on the full 678-patient cohort. For frozen-feature trees on Derm7pt or TCGA, no reported result was found on Lacuna, so we produce those numbers ourselves.
The strongest end-to-end competitor on Derm7pt is TFormer, which reports 77.99% average accuracy and 80.03% diagnostic accuracy on the official split (Zhang & Xie). We rerun it on our subsamples. On TCGA the end-to-end competitor is HEALNet; it reports only relative gains, so no reported absolute result was found on Lacuna for HEALNet per cohort. For deep fusion with representation-level regularization, we use multimodal feature MixUp and channel masking with minority-class noise in the style of RepAugment, both applied to the same fusion head. The late-fusion baseline is a sum of per-modality logits with modality-specific learning rates, following Jha et al..
The key ablations isolate the ingredient. We compare $\rho=0$, which is the same head with no recombination, uniform against risk-matched recombination, and recombination applied to an additive head versus an interaction head. We also include a tuned convex blend of late-fusion and deep-fusion predictions; this tests whether recombination is anything more than shrinkage toward additivity.
Every method uses the same frozen encoder features and the same splits. Each gets an equal hyperparameter budget of 30 trials, chosen by nested cross-validation inside the training subsample, and the same number of seeds. Our method counts as winning only if it beats both the best classical baseline and the best end-to-end baseline on discrimination, with a seed-paired confidence interval excluding zero, at $N \le 200$, without losing calibration. At full $N$, we only require it to match deep fusion.
Why not just…?
Why not just fit XGBoost on frozen concatenated features? This is the real threat, and the evidence of Alhuzali et al. and Liu et al. says it often wins at small $N$. Against it, Nelson et al. found that XGBoost combiners still overfit without synthetic augmentation, and TabMixer found that integrating features inside the backbone beat late concatenation. Our subsampling curves settle the question directly. If trees match recombined fusion at every $N$, we will report that cross-modal interactions are not learnable below that size, which is itself a useful finding for practitioners.
Why not just use late fusion with adapted learning rates? Late fusion is what our method converges to at $\rho=1$. If the best $\rho$ is always 1, interactions add nothing at these sample sizes and the method reduces to the Jha et al. baseline. The published gains of CA-MLIF and HEALNet over late fusion suggest interactions do matter at moderate $N$. The $\rho$ sweep tests whether that remains true as $N$ shrinks.
Why not just blend late-fusion and deep-fusion predictions, or penalize the interaction weights? This is the honest concern that recombination is merely shrinkage done through data. The blend ablation tests it head-on. Recombination can only add something beyond the blend if the expressive head, trained on many label-consistent tuples, learns better per-modality features than separate unimodal heads do. If the blend matches our method, we will say so.
Why not just use MixUp or RepAugment-style feature noise? MixUp-MIL shows that interpolating across subjects creates unrealistic features. Feature noise adds variety within a modality but no new cross-modal combinations. Both are run on the identical head, so the comparison is direct.
Why not just fine-tune encoders with LoRA? Dutt et al. show that parameter-efficient tuning beats full fine-tuning at low data, but that addresses the encoders, not the fusion head. We add one Derm7pt arm with LoRA-adapted encoders to check that recombination still helps on top of adapted features.
Proposed Experiments
The main experiment covers four settings. Derm7pt uses three modalities with the official split, subsampled to 50, 100, 200 and the full training set. TCGA uses slides plus omics on the BLCA, BRCA, KIRP and UCEC survival cohorts as in HEALNet, subsampled the same way and evaluated by C-index. Cyst-X uses the multi-stream MRI risk task, evaluated by QWK and AUC, if access permits. MIMIC-III in-hospital mortality uses tabular data plus time series, subsampled to a few hundred admissions, as a non-imaging check. Each cell runs over at least ten seeds with stratified subsampling. Metrics are discrimination, expected calibration error, and minority-class recall, the last because of the danger zone reported by Nelson et al..
Three decisive checks follow. First, the best $\rho$ should decrease monotonically with $N$ on at least three of the four settings; if it is flat at 1, the interaction hypothesis fails. Second, risk-matched recombination should beat uniform recombination at full $N$ while matching it at $N=50$, which would show that the matching preserves real dependence. Third, on a semi-synthetic control, we inject a known cross-modal interaction into Derm7pt labels, for example by making the diagnosis depend on agreement between metadata and an image attribute. Uniform recombination at high $\rho$ should then measurably hurt, and the pairing gap should detect the dependence. This confirms that both the tradeoff and the diagnostic behave as claimed.
A final test covers missing modalities. Modalities are dropped at random at test time, and we ask whether recombination with modality dropout matches HEALNet's update-skipping robustness at small $N$.
References
- TFormer: A throughout fusion transformer for multi-modal skin lesion diagnosis
- A Novel Perspective for Multi-Modal Multi-Label Skin Lesion Classification
- HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data
- CA-MLIF: Cross-Attention and Multimodal Low-Rank Interaction Fusion Framework for Tumor Prognostic Prediction
- Predicting Sign of Depression via Using Frozen Pre-trained Models and Random Forest Classifier
- Automatic risk prediction of intracranial aneurysm on CTA image with convolutional neural networks and radiomics analysis
- Gaussian Meta-Space Augmentation for Stacking Ensembles in Multimodal IPMN Risk Stratification
- MixUp-MIL: Novel Data Augmentation for Multiple Instance Learning and a Study on Thyroid Cancer Diagnosis
- Improving information fusion on multimodal clinical data in classification settings
- RepAugment: Input-Agnostic Representation-Level Augmentation for Respiratory Sound Classification
- TabMixer: Noninvasive Estimation of the Mean Pulmonary Artery Pressure via Imaging and Tabular Data Mixing
- Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity