Knowledge Transfer Scaling Laws for 3D Medical Imaging

Ho Hin Lee, Dongna Du, Chu Wang, Yuankai Huo, Shi Gu, James C. Gee, Yifan WuarXiv 2026 · arXiv

TL;DR

Pretraining 3D medical foundation models across diverse modalities (CT, MRI, PET) usually relies on guesswork or simple heuristics to mix data. This paper introduces a mathematically principled transfer-aware scaling law that models domain-specific scaling rates and asymmetric cross-domain knowledge transfer. By optimizing data mixtures using small-scale proxy runs, the authors achieve up to a 58% relative improvement in pretraining efficiency and significantly boost downstream clinical task performance.

What Problem Does This Paper Address?

Building a unified 3D medical foundation model requires training on highly heterogeneous datasets spanning different modalities (CT, MRI, PET) and anatomies (brain, abdomen, head/neck). However, given a fixed training token or compute budget, a fundamental question remains open: How much data should be sampled from each imaging domain?

Currently, researchers rely on simple heuristics, such as uniform splitting or sampling proportionally to dataset size. These approaches treat all imaging domains as interchangeable. In reality, some modalities are harder to learn than others, and learning features in one domain (e.g., abdominal CT) might drastically help performance in another (e.g., abdominal MRI) more than vice versa. This paper replaces heuristic data mixing with an analytical, compute-optimal allocation strategy.

Methods

The authors use Masked Autoencoding (MAE) reconstruction loss as a controlled proxy objective to measure representation quality across domains. They formulate data allocation as a scaling-law optimization problem based on two core regularities:

  1. Domain-Specific Self-Scaling: Each domain $i$ scales at its own rate, modeled by a power-law exponent $\beta_i$. A larger $\beta_i$ means the domain continues to improve rapidly with more data, while a smaller one indicates fast saturation.
  2. Directed Cross-Domain Transfer: Knowledge transfer is directional and quantified by an asymmetric transfer matrix $\tau_{ij}$. This matrix dictates how effectively training on source domain $j$ acts as "effective samples" for target domain $i$.
Figure 1: Observations motivating the transfer-aware scaling law. (a) Per-domain MAE loss follows Li = Ei + Ci/T βi with domain-specific exponents varying 3.5× (β = 0.21 for BRAIN T1 to β = 0.76 for HN PET), indicating that domains benefit from additional data at fundamentally different rates. (b) Cross-domain transfer obeys the same power-law form with a directional discount τij: ABD CT transfers strongly to ABD MRI (τ=0.56) but weakly to HN PET (τ=0.04), and the reverse ABD MRI →ABD CT (τ=0.42, dashed) is weaker than the forward direction. (c) Full 6×6 transfer matrix τij (from source j to target i) estimated from pure-domain proxy runs. The matrix is asymmetric (τij ̸= τji) and reveals hub-and-island structure: ABD CT (hub) transfers broadly to most domains, while HN PET (island) is largely isolated.

By conducting small-scale, pure-domain proxy runs, the authors fit these parameters and define a closed-form surrogate to predict multi-domain loss. They then apply gradient-based constrained optimization (specifically Sequential Least Squares Programming, or SLSQP) to solve for the optimal data mixture weights under any given compute budget.

Datasets

  • Pretraining Domains: Six distinct 3D medical imaging datasets subsampled from public collections to simulate realistic data imbalances: Abdominal CT (AMOS 2022), Abdominal MRI (AMOS 2022), Brain MRI-T1 (BraTS 2023), Brain MRI-T2 (BraTS 2023), Head-and-Neck CT (HECKTOR 2025), and Head-and-Neck PET (HECKTOR 2025).
  • Downstream Evaluation: The pretrained models were validated on six supervised tasks:
  • Segmentation: Abdominal organ segmentation (AMOS 2022), brain tumor segmentation (BraTS 2023), and head-and-neck lesion segmentation (HECKTOR 2025, BHSD 2024).
  • Classification: Structural brain MRI aging/cognitive impairment (OASIS-1), functional brain PET (SynBrain-PET), and 3D abdominal organ recognition (OrganMNIST3D).

Key Insights and Ideas

  • The "Hub-and-Island" Structure: The estimated transfer matrix reveals a fascinating data topology. Highly transferable domains emerge as hubs (e.g., Abdominal CT), which broadly benefit multiple target domains and amplify their effective training budgets. Conversely, specialized modalities act as islands (e.g., Head-and-Neck PET); they receive virtually zero transfer benefit from other domains and require direct, targeted data investment.
  • Asymmetric Transfer: Knowledge transfer is a one-way street. For example, Abdominal CT transfers strongly to Abdominal MRI ($\tau = 0.56$), but the reverse direction is notably weaker ($\tau = 0.42$).
  • Heterogeneous Scaling Exponents: Scaling exponents vary by up to 3.5× across domains ($\beta = 0.21$ for Brain T1 vs. $\beta = 0.76$ for HN-PET). This proves that assuming a single, shared scaling exponent across medical modalities—as common in natural language processing scaling laws—is fundamentally insufficient.

Key Results

  • 58% Efficiency Gap: In intermediate data budgets, the transfer-aware allocation strategy outperforms standard data-proportional sampling by up to 58% in relative MAE validation loss.
  • Robust Extrapolation: When trained on small budgets ({10K, 50K, 100K} volume draws), the scaling law accurately predicts performance at an unseen, larger budget (200K) with an exceptional Pearson correlation of $r = 0.989$.
Scaling law validation, showing mixture strategy comparisons, extrapolation accuracy, and strong per-domain correlation.
  • Downstream Performance Gains: The optimized pretraining recipe consistently translates to superior downstream performance. For instance, on the BraTS Brain T1 segmentation task, the transfer-aware mixture improved the Dice score to 55.1, compared to 50.4 for data-proportional pretraining and 45.2 for training from scratch.

Limitations and Future Work

  • Linear Composition Assumption: The framework assumes that transfer contributions from different source domains accumulate linearly. This ignores potential synergistic or destructive multi-domain interactions that might occur when mixing three or more sources simultaneously.
  • Static Transfer Matrix: The transfer matrix $\tau$ is treated as a constant across budget scales. However, as foundation models scale up significantly, the transfer structure might evolve as models shift from learning low-level features to highly domain-specific representations. Modeling $\tau$ as a budget-dependent function is an open avenue for future research.

Summary updated

Abstract

Vision foundation models are increasingly moving beyond 2D to volumetric domains such as 3D medical imaging, where unified pretraining across different imaging modalities (i.e. CT, MRI, and PET) could provide foundational models for diverse clinical tasks. However, training such models requires mixing heterogeneous imaging domains, and current mixture strategies remain largely heuristic. In this work, we observe that different medical imaging domains scale at variable rates during pretraining, and knowledge transfer between domains is strongly asymmetric: training on one domain can substantially improve another, but the reverse may be much weaker. Interestingly, both MAE reconstruction loss and cross-domain transfer follow predictable power-law trends with domain-specific behaviors. Motivated by these findings, we formulate data allocation as a scaling-law optimization problem. The derived allocations reveal an interpretable hub-and-island structure: highly transferable domains emerge as hubs that benefit many others and deserve strategic allocation, while isolated domains act as islands requiring direct investment. Empirically, transfer-aware allocation outperforms data-proportional sampling by up to 58% and generalizes well to unseen budgets with r=0.989. Downstream validation on disease classification and organ/lesion segmentation further confirms that the derived transfer-aware mixtures provide stronger pretrained representations for clinical 3D medical imaging tasks.
BibTeX
@misc{lee2026knowledge,
  author = {Ho Hin Lee and Dongna Du and Chu Wang and Yuankai Huo and Shi Gu and James C. Gee and Yifan Wu},
  title = {{Knowledge Transfer Scaling Laws for 3D Medical Imaging}},
  year = {2026},
  eprint = {2605.06859},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.06859},
}
Figure