Cross Dimensional Weight Inflation for 3D Vision

This direction is split into other directions: cross-dimensional weight transfer. Its last version stays readable and citable here.

Version 1 of 192 papers137 conceptsfrozen 2026-10-04

The central challenge in 3D computer vision—whether processing medical CT scans, LiDAR point clouds, or synthetic shapes—is the chronic scarcity of labeled data. While 2D image datasets like ImageNet or LAION contain billions of samples, 3D datasets are often orders of magnitude smaller due to the high cost of expert annotation and the complexity of data acquisition.

To bridge this gap, a recurring research direction has emerged: Weight Inflation. Instead of training 3D models from scratch, researchers "lift" the weights of highly optimized 2D models into the third dimension. This allows 3D architectures to inherit the rich visual priors and semantic reasoning capabilities developed on massive 2D datasets.

The Mechanics of Inflation

The core idea behind inflation is to transform a 2D convolutional kernel (typically $k \times k$) into a 3D kernel ($k \times k \times d$). As Yang Song and colleagues demonstrate in 3D GCA-Net, this can be achieved through "pseudo-3D" initialization, where 2D filters are expanded into 3D kernels with a minimal depth dimension.

There are two primary ways to perform this expansion:

  1. Center-Slicing: Placing the 2D weights in the center of the 3D kernel and padding the rest with zeros.
  2. Replication: Repeating the 2D weights along the depth dimension.

In INN: Inflated Neural Networks, Rodney LaLonde and Ulas Bagci emphasize a crucial detail: when replicating weights, one must scale the values by the inverse of the depth (e.g., dividing by $d$). This ensures that the total activation energy remains consistent with the original 2D model, preventing the network's signals from exploding as they pass through the newly added dimension.

Why 2D Priors Transfer to 3D

It might seem counterintuitive that a model trained to recognize cats and dogs on flat pixels could help segment a prostate in a 3D MRI. However, the intuition rests on the fact that low-level visual features are dimension-agnostic. Edges, textures, and local gradients look remarkably similar whether they are viewed as a 2D slice or a 3D volume.

Chenfeng Xu and the authors of Image2Point show that this transferability extends even to sparse coordinate data like point clouds. By inflating 2D weights to process voxelized representations, they achieved an 11x speedup in training. This suggests that the "visual logic" learned from dense 2D grids—such as how to group local features into global structures—is a fundamental geometric prior that the 3D domain can exploit.

From CNNs to Transformers

The trend has recently shifted from inflating convolutional kernels to inflating Vision Transformers (ViTs). In Simple3D-Former, Yi Wang 0076 demonstrates a "minimalist" approach: inflating the 2D patch embedding layer into a 3D voxel embedding.

This strategy reached its zenith with Uni3D, where Junsheng Zhou scaled 3D models to one billion parameters. By initializing a standard ViT with 2D weights and aligning it with the image-text feature space of models like CLIP, Uni3D allows 3D models to inherit not just visual features, but semantic "knowledge" about the world. This enables zero-shot 3D classification, where a model can identify a 3D object it has never seen before simply by leveraging its 2D-trained understanding of the object's name and appearance.

Handling Anisotropy and Stability

A common hurdle in 3D medical imaging is anisotropy—where the resolution within a slice is much higher than the resolution between slices. Siqi Liu 0001 addressed this in AH-Net by using specialized anisotropic convolutions that preserve the 2D feature hierarchy while gradually integrating 3D context.

Furthermore, introducing 3D operations can sometimes destabilize a pre-trained 2D backbone. To counter this, Jiancheng Yang and the team behind Asymmetric 3D Context Fusion proposed initializing 3D context operators as identity matrices. This ensures the model begins training with the exact behavior of its 2D-pretrained parent, only incorporating volumetric information as the loss function demands it.

The Impact on the Field

The shift toward 2D-to-3D transfer learning has fundamentally changed the "small data" regime of 3D vision. As noted in the survey by Vishal Thengane on Foundational Models for 3D Point Clouds, "piggybacking" on the success of 2D models is no longer just a shortcut—it is the standard for achieving state-of-the-art performance. By treating 3D data as a spatial extension of 2D visual patterns, researchers have unlocked the ability to train deep, high-capacity models on datasets that were previously considered too small to support them.

Papers in this version

Showing 60 of 92 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_dimensional_weight_inflation_for_3_b1a4df2d,
  title = {Cross Dimensional Weight Inflation for 3D Vision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-dimensional-weight-inflation-for-3d-vision/rdr_b1a4df2dbc784b8e99832a9b890ee62b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_dimensional_weight_inflation_for_3_b1a4df2d_v1,
  title = {Cross Dimensional Weight Inflation for 3D Vision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-dimensional-weight-inflation-for-3d-vision/rdr_b1a4df2dbc784b8e99832a9b890ee62b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}