Deep learning thrives on data abundance. In two-dimensional (2D) computer vision, models benefit from millions of curated images (like ImageNet) or massive internet-scale image-text datasets. However, when moving to higher dimensions—such as 3D medical imaging (CT/MRI scans), video, hyperspectral data, or point clouds—we hit a wall. Volumetric and multi-channel data are notoriously scarce, expensive to annotate, and computationally heavy to train from scratch.
As noted by Jiancheng Yang et al. in Advances and challenges in medical 3D computer vision, this "small data" challenge severely limits the performance of native 3D architectures. Training a 3D network from scratch often leads to severe overfitting or requires aggressive volume compression that discards critical spatial information, a problem highlighted by Hao Chen et al. in Unifying Structure Analysis and Surrogate-driven Function Regression for Glaucoma OCT Image Screening. In fact, Yu Zhang et al. demonstrated in 2D Convolutional Neural Networks for 3D Digital Breast Tomosynthesis Classification that standard 2D networks leveraging transfer learning can frequently outperform native 3D networks trained on limited volumetric data.
This raises a fundamental question: How can we give a model the spatial awareness of a 3D network while inheriting the rich, general-purpose feature representations learned by 2D models? The answer lies in cross-dimensional weight transfer, commonly known as weight inflation.
The Intuition Behind Weight Inflation
Why should a model trained to recognize everyday objects help a network segment a kidney tumor or track an object in a hyperspectral video? The core intuition is that visual primitives—such as edges, textures, boundaries, and spatial hierarchies—are fundamentally dimension-agnostic. A 3D volume or a multi-channel cube is composed of these exact same primitives.
The simplest way to bridge this dimensional gap is to take a pre-trained 2D convolutional kernel (e.g., $3 \times 3$) and replicate its weights along a newly introduced third dimension to form a 3D kernel (e.g., $3 \times 3 \times 3$). As explored by Chenfeng Xu et al. in Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models, this cross-dimensional inflation allows models pre-trained on dense 2D images to initialize weights for 3D point clouds or voxelized data, preserving learned spatial features and accelerating training speeds by up to 11x.
Similarly, Junlin Hou et al. in CMC-COV19D: Contrastive Mixup Classification for COVID-19 Diagnosis used depth-wise weight inflation to allow volumetric models to inherit robust feature extractors from natural images, bypassing the need to train data-hungry 3D networks from scratch. This concept even extends across vision-language boundaries: Léo Milecki demonstrated in MEDIMP: Medical Images and Prompts for renal transplant representation learning that replicating weights from a 2D vision-language foundation model across the depth dimension provides a highly effective initialization for 3D medical image encoders, a strategy further validated by Léo Milecki et al. in MEDIMP: 3D Medical Images and clinical Prompts for renal transplant representation learning.
Advanced Inflation and Decomposition Strategies
While uniform replication is straightforward, it treats every slice along the new dimension identically, which can fail to capture complex geometric structures. Researchers have developed more sophisticated strategies to inject geometric bias or decompose spatial context.
Geometric and Gaussian Bias
Instead of copying weights uniformly, Daniel Kienzle et al. in COVID Detection and Severity Prediction with 3D-ConvNeXt and Custom Pretrainings proposed distributing the pre-trained 2D weights along the new axis using a Gaussian distribution. This introduces a geometric prior that emphasizes the center of the volumetric window, significantly improving the network's ability to capture 3D spatial structures.
Orthogonal Decomposition (ACS Convolutions)
Another elegant approach is to split the 2D kernels to process orthogonal views of a 3D volume. Jiancheng Yang et al. introduced Axial-Coronal-Sagittal (ACS) convolutions in Reinventing 2D Convolutions for 3D Images. By mapping 2D kernels to multiple orthogonal planes within a 3D space, the model gains 3D spatial awareness while directly recycling 2D weights.
This strategy is echoed by Jinzheng Cai et al. in Deep Volumetric Universal Lesion Detection using Light-Weight Pseudo 3D Convolution and Surface Point Regression and Won-Woo Cho et al. in Slice and Conquer: A Planar-to-3D Framework for Efficient Interactive Segmentation of Volumetric Images, demonstrating that planar-to-3D weight recycling mitigates data scarcity across diverse volumetric tasks.
Identity Initialization
To prevent the disruption of pre-trained features at the start of training, Jiancheng Yang et al. in Asymmetric 3D Context Fusion for Universal Lesion Detection proposed initializing 3D context operators with an identity matrix. This ensures the network behaves exactly like its 2D-pretrained baseline on day one, allowing it to incrementally integrate volumetric information without destroying its inherited knowledge.
Adapting Transformers and Neural Operators
The cross-dimensional transfer paradigm is not limited to convolutional networks; it applies equally to Vision Transformers (ViTs) and neural operators.
For instance, Lulu Gai et al. showed in Comparing CNN-based and transformer-based models for identifying lung cancer: which is more effective? that core transformer blocks can be initialized with 2D pre-trained weights while training input-specific components (like patch embeddings) from scratch to handle the extra dimension. To explicitly capture cross-slice dependencies in 2D foundation models like SAM, Hao Li 0009 et al. in Promise: Prompt-driven 3D Medical Image Segmentation Using Pretrained Image Foundation Models inserted lightweight volumetric adapter modules using depth-wise convolutions directly into frozen 2D transformer layers.
Remarkably, this principle even works across physical dimensions in scientific computing. AmirPouya Hemmasian et al. developed a strategy in Pretraining a Neural Operator in Lower Dimensions that pretrains a Fourier Neural Operator on computationally cheap 1D partial differential equations (PDEs) and replicates those weights across multiple axes to initialize higher-dimensional 2D or 3D models, exploiting structural symmetry to transfer physical priors.
Expanding the Channel Space
Cross-dimensional transfer also solves the problem of adding new input modalities or spectral bands to standard three-channel (RGB) architectures.
When adapting an RGB model for depth-sensing or multi-modal tasks, researchers face the challenge of initializing the extra channels. Nawid Jamali et al. in Robot Bed-Making: Deep Transfer Learning Using Depth Sensing of Deformable Fabric showed that triplicating single-channel depth data into a three-channel format allows it to immediately exploit RGB-pretrained feature extractors. Alternatively, Xiaoqi Zhao et al. in A Single Stream Network for Robust and Real-Time RGB-D Salient Object Detection and Qian Chen et al. in RGB-D Salient Object Detection via 3D Convolutional Neural Networks inflated 2D kernels to process stacked modality tensors directly, preserving the generalization power of RGB pre-training.
For non-standard modalities like near-infrared (NIR) or hyperspectral imaging, weight copying provides an elegant initialization. In Agriculture-Vision: A Large Aerial Image Database for Agricultural Pattern Analysis, Mang Tik Chiu et al. extended an RGB model to a fourth NIR channel by duplicating the pre-trained red channel weights. Similarly, Shaheer Mohamed et al. in Spectral-Enhanced Transformers: Leveraging Large-Scale Pretrained Models for Hyperspectral Object Tracking inflated the input layers of large-scale RGB-pretrained transformers to accommodate dozens of narrow spectral bands.
Finally, Pranav Gupta et al. demonstrated in OLAF: A Plug-and-Play Framework for Enhanced Multi-object Multi-part Scene Parsing that averaging the weights of pre-trained filters to initialize new structural channels, combined with a training warm-up phase, allows networks to stably incorporate new data modalities without discarding learned knowledge.
Key Foundational Work
Cardiac Segmentation of LGE MRI with Noisy Labels (2019) — contributed 2 concept elements to this direction.
3-D Convolutional Encoder-Decoder Network for Low-Dose CT via Transfer Learning From a 2-D Trained Network (2018) — contributed 2 concept elements to this direction.
SurfConv: Bridging 3D and 2D Convolution for RGBD Images (2018) — contributed 1 concept elements to this direction.