Learnable Weight Initialization for Volumetric Medical Image Segmentation

Shahina Kunhimon, Abdelrahman Shaker, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan

In the world of deep learning, how you start often determines where you finish. For volumetric medical image segmentation—tasks like identifying tumors or organs in 3D CT scans—most researchers focus on building bigger, better architectures. However, a new paper by Shahina Kunhimon and colleagues suggests we’ve been overlooking a fundamental piece of the puzzle: weight initialization.

TL;DR

The authors propose a learnable, data-dependent weight initialization scheme that uses self-supervised learning on the same small-scale dataset intended for the final task. By teaching models to understand 3D structure and context before the actual training begins, they achieve state-of-the-art results on multi-organ and cancer segmentation without needing massive external datasets.

What Problem Does This Paper Address?

Deep learning models are notoriously data-hungry, but high-quality, annotated 3D medical images are rare and expensive to produce. To compensate, researchers often use "hybrid" architectures that combine the local feature extraction of CNNs with the global context of Transformers (like UNETR or Swin-UNETR).

Despite these architectural advances, most models still start their "lives" with weights drawn from standard, data-independent distributions like Xavier or Kaiming initialization. These methods treat the model as a blank slate, ignoring the inherent 3D anatomical structures present in medical data. While large-scale pre-training on thousands of external scans is a common workaround, it introduces domain gaps (e.g., pre-training on CT but testing on MRI) and massive computational overhead.

This paper asks: Can we learn a better starting point using only the limited data we already have?

Methods

The authors introduce a two-step framework that replaces static initialization with a learnable one.

Step 1: Self-Supervised Initialization

Before the model ever sees a segmentation label, it undergoes a "warm-up" phase using two self-supervised tasks designed to capture the nature of volumetric data:

  1. Structural Consistency (Order Prediction): The input volume is chopped into sub-volumes and shuffled. The encoder must predict the original order of these pieces, forcing it to learn the global anatomical layout (e.g., where the liver sits relative to the spine).
  2. Contextual Consistency (Voxel Reconstruction): Parts of the volume are masked out. The decoder must reconstruct these missing regions, forcing the model to understand local textures and organ boundaries.
Figure 2: Overview of our proposed approach: To learn weight initialization using self-supervised tasks defined by the volumetric nature of the medical data. In the early stage of training (Step-1), we define the order prediction task within the encoder latent space, while simultaneously the decoder has to reconstruct the missing organs from masked & shuffled input. The masked & shuffled input is the result of our transformation module with 4 stages: depth-wise rearranging, partitioning into equal size sub-volumes, random shuffling of sub-volumes for the order prediction objective, and finally masking shuffled volume for the reconstruction objective. This allows the model to learn structural and contextual consistency about the data that provides an effective initialization for the segmentation task (Step-2). Our approach does not rely on any extra data and therefore remains as computationally effective as the baseline while enhancing the segmentation performance.

Step 2: Supervised Training

The weights learned in Step 1 are transferred to the segmentation task. The model is then trained normally using standard losses like Dice and Cross-Entropy. Because the model already "understands" 3D anatomy, it converges to a better solution.

Datasets

The approach was validated on two primary benchmarks:

  • Synapse Multi-organ CT: 30 abdominal CT scans with 8 organ labels. This is a classic "small data" challenge, with only 18 samples used for training.
  • Decathlon Lung Dataset: 63 CT volumes focused on binary lung cancer segmentation.
  • BTCV: Used for additional comparison against large-scale pre-trained models.

Key Insights and Ideas

  • Initialization Sensitivity: The authors show that hybrid models like UNETR are incredibly sensitive to their starting weights. Simply switching from Kaiming to Truncated Normal initialization can significantly swing performance, yet most papers treat this as a fixed hyperparameter.
  • Small Data, Big Impact: You don't need 5,000 external scans to "pre-train." By using the target dataset's own structural properties, the model gains inductive biases that are perfectly aligned with the downstream task.
  • Better for Small Organs: The learnable initialization showed the most significant gains in segmenting small, difficult organs like the aorta, gallbladder, and pancreas, where standard models often struggle with false positives.

Key Results

The proposed method consistently outperformed baseline models across multiple architectures (UNETR, UNETR++, MedNeXt).

  • Synapse Dataset: On the UNETR architecture, the learnable initialization boosted the average Dice score from 77.17% to 79.73%.
  • State-of-the-Art: When integrated with UNETR++, the method achieved an average Dice of 87.74%, surpassing existing SOTA methods.
  • Efficiency: The authors demonstrated that their method, using only the local dataset, performs favorably compared to Swin-UNETR models pre-trained on massive, 5,000+ scan cohorts.

Qualitatively, the results show much cleaner boundaries and a significant reduction in "hallucinated" organ segments (false positives), as seen in the comparison below:

Figure 3: Qualitative results for Synapse dataset on SOTA segmentation networks: The proposed data-dependent initialization scheme, when integrated with different segmentation networks, improves the overall segmentation performance by accurately segmenting the organs and delineating organ boundaries. Organs are shown in the legend below the example images. Abbreviations are as follows: Spl: spleen, RKid: right kidney, LKid: left kidney, Gal: gallbladder, Eso: esophagus, Liv: liver, Sto: stomach, Aor: aorta, IVC: inferior vena cava, PSV: portal and splenic veins, Pan: pancreas, RAG: right adrenal gland, and LAG: left adrenal gland. Best Viewed zoomed in.

Limitations and Future Work

The primary limitation is the additional computational cost. Adding a self-supervised "Step 1" increases the total training time compared to a single-stage supervised run. However, the authors argue this cost is negligible compared to the weeks of training required for large-scale external pre-training.

Future work could explore how these self-supervised objectives might be tuned for different modalities, such as MRI or Ultrasound, where the "structural cues" might differ significantly from CT scans.

Reference: Kunhimon, S., et al. "Learnable Weight Initialization for Volumetric Medical Image Segmentation." arXiv:2306.09320v4, 2024. GitHub Link

Summary updated

Abstract

Hybrid volumetric medical image segmentation models, combining the advantages of local convolution and global attention, have recently received considerable attention. While mainly focusing on architectural modifications, most existing hybrid approaches still use conventional data-independent weight initialization schemes which restrict their performance due to ignoring the inherent volumetric nature of the medical data. To address this issue, we propose a learnable weight initialization approach that utilizes the available medical training data to effectively learn the contextual and structural cues via the proposed self-supervised objectives. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach, leading to state-of-the-art segmentation performance. Our proposed data-dependent initialization approach performs favorably as compared to the Swin-UNETR model pretrained using large-scale datasets on multi-organ segmentation task. Our source code and models are available at: https://github.com/ShahinaKK/LWI-VMS.
BibTeX
@article{kunhimon2024learnable,
  author = {Shahina Kunhimon and Abdelrahman Shaker and Muzammal Naseer and Salman Khan and Fahad Shahbaz Khan},
  title = {{Learnable Weight Initialization for Volumetric Medical Image Segmentation}},
  journal = {Artif. Intell. Medicine},
  year = {2024},
  doi = {10.1016/j.artmed.2024.102863},
}
Figure