Initializing Video Models with Image Pre-training

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 145 papers63 conceptsfrozen 2026-10-04

Video recognition models face a fundamental data paradox: while video contains significantly more information than static images, high-quality labeled video datasets are orders of magnitude smaller and harder to collect than image datasets like ImageNet. To overcome this "scarcity" bottleneck, researchers have developed a suite of techniques to "inflate" the knowledge trapped in 2D image-based models into 3D spatio-temporal architectures.

The core intuition is that a model shouldn't have to learn what an object looks like (spatial features) and how it moves (temporal features) simultaneously from scratch. Instead, we can give a model a "semantic head start" by initializing it with weights from a pre-trained image model, allowing the training process to focus almost entirely on motion dynamics.

The Inflation Strategy

The most straightforward way to bridge the gap between 2D and 3D is weight inflation. In a standard 2D Convolutional Neural Network (CNN), a filter might be $3 \times 3$. In a 3D video network, that filter becomes $3 \times 3 \times T$, where $T$ is the temporal dimension.

As demonstrated by Ze Liu and colleagues in the Video Swin Transformer, one can initialize these 3D kernels by simply duplicating the 2D weights along the temporal axis. This ensures that, at the start of training, the 3D model behaves exactly like its 2D counterpart applied to each frame independently. This "inflation" technique allows models to benefit from massive image-based pre-training, leading to faster convergence and higher accuracy on benchmarks like Kinetics-400.

Anchoring the Spatial Prior

While simple duplication is effective, it can sometimes lead to a "discrepant initialization" where the model struggles to differentiate between static appearance and motion. To solve this, researchers have proposed more surgical initialization methods:

  • Center-Slice Initialization: Instead of duplicating weights, the pre-trained 2D weights are placed only in the middle temporal slice of the 3D kernel, with all other slices set to zero. This approach, defended in In Defense of Image Pre-Training for Spatiotemporal Recognition, anchors the model to a static appearance prior while giving the surrounding temporal slices a "blank slate" to learn motion offsets.
  • Tail Initialization: In generative tasks, such as those explored in OD-VAE, weights are mapped to the final temporal element of a 3D kernel. This provides a strong spatial baseline that the model can then extend backward in time to learn consistency.

Architectural Factorization

Another way to make transfer learning easier is to design the video architecture to be "2D-friendly" from the start. Zhaofan Qiu and Tao Mei introduced Pseudo-3D (P3D) Residual Networks, which decompose a 3D convolution into a 2D spatial convolution followed by a 1D temporal convolution.

This factorization means the spatial component can be directly initialized with standard ImageNet weights without any inflation or modification. Similarly, Yanbin Hao and the team behind PosMLP-Video use specialized gating units that separate spatial and temporal modeling. By substituting temporal units with residual connections during an initial phase, they allow the model to inherit spatial weights while temporal-specific parameters are fine-tuned later.

Scaling to High Dimensions

As video models move toward higher frame rates, the gap between 2D and 3D grows. Yixuan Li and Wei Li addressed this in their work on the F-16 model, which processes 16 frames per second. To transition from a static image aligner to a high-dimensional temporal aligner, they used block matrix decomposition. This mathematical mapping preserves the learned visual knowledge of the image model even as the input dimensionality increases, preventing the training instability that often occurs when moving to dense video data.

Why It Works: The Semantic Head Start

The success of these methods across diverse tasks—from action recognition to video generation and quality assessment—stems from the fact that spatial features are largely universal. A "car" or a "face" has the same basic visual structure whether it is in a still photo or a 60 FPS video.

By using strategies like supervision transfer (where a 2D model guides a 3D model, as proposed by Ali Diba in T3D), researchers ensure that the model doesn't waste its limited video training data learning that edges and textures exist. Instead, the model starts with a sophisticated understanding of the world's appearance and can dedicate its entire capacity to understanding how that world changes over time.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_initializing_video_models_with_image_pre_cf2f38fd,
  title = {Initializing Video Models with Image Pre-training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initializing-video-models-with-image-pre-training/rdr_cf2f38fdda6347ad9d1662ed86967642},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_initializing_video_models_with_image_pre_cf2f38fd_v1,
  title = {Initializing Video Models with Image Pre-training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initializing-video-models-with-image-pre-training/rdr_cf2f38fdda6347ad9d1662ed86967642/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}