In the development of medical AI, 2D imaging—such as X-rays and pathology slides—has long benefited from "ImageNet pretraining." By starting with a model that already understands shapes, textures, and edges from millions of natural images, researchers can achieve high performance even with small medical datasets.
However, 3D medical imaging, such as Computed Tomography (CT) and Magnetic Resonance Imaging (MRI), has historically lacked a "3D ImageNet." Training 3D convolutional neural networks (CNNs) from scratch requires massive amounts of labeled volumetric data, which is notoriously difficult and expensive to acquire. To solve this, researchers have turned to an unexpected source of data: natural video.
The Analogy of Time and Depth
The core intuition behind this research direction is a structural analogy between the temporal dimension of a video and the spatial depth of a volumetric scan.
Mathematically, a video is a stack of 2D frames ordered by time ($H \times W \times Time$). A CT scan is a stack of 2D slices ordered by their physical position in the body ($H \times W \times Depth$). As Jie Yang and colleagues noted in Class-Aware Adversarial Lung Nodule Synthesis in CT Images, mapping spatial depth to the temporal dimension allows models to treat the "flow" of anatomy across slices as if it were the "motion" of objects across frames.
When a camera moves through a scene, objects change shape, appear, and disappear in a predictable, continuous manner. Similarly, as a radiologist scrolls through a CT scan, an organ like the liver or a vessel like the aorta appears to "move" or shift its morphology. By pretraining on natural videos (like YouTube clips or the Kinetics dataset), a model learns the fundamental physics of continuity and motion, which then generalizes to the spatial continuity of human anatomy.
Transferring Motion to Morphology
This cross-domain transfer learning works because the features required to recognize a person running are surprisingly similar to those needed to identify a lesion in a lung. Both tasks require the model to integrate information across multiple "frames" to understand the 3D structure of an object.
In AppendiXNet, Pranav Rajpurkar, Allison Park, and their team demonstrated that a 3D model pretrained on natural videos could diagnose appendicitis with high accuracy using fewer than 500 CT scans. This is a remarkably small dataset for a 3D task, suggesting that the model's "understanding" of how structures evolve across slices was largely inherited from its video pretraining.
Similarly, Alexander Ke and Serena Yeung-Levy showed in Video pretraining advances 3D deep learning on chest CT tasks that this spatiotemporal initialization provides a more robust starting point than traditional 2D weights, specifically because it captures the inter-slice relationships that 2D models ignore.
Zero-Shot Propagation and Interactive Segmentation
The utility of video-based models extends beyond simple classification. Recent work has applied video foundation models directly to medical segmentation. In Interactive 3D Medical Image Segmentation with SAM 2, Chuyun Shen and Wenhao Li utilized Meta’s Segment Anything Model 2 (SAM 2)—originally designed for video object tracking—to segment medical volumes.
By treating the slices of a scan as video frames, the model can "track" an organ through the body. A clinician can annotate a single slice, and the model propagates that mask through the entire 3D volume in a zero-shot manner. This works because the model’s learned representation of object permanence and deformation in natural video translates effectively to the anatomical morphology of organs.
Beyond Static Scans: Medical Video Analysis
While much of this research focuses on using video to understand static 3D volumes (like CT), the same principles apply to actual medical videos, such as echocardiograms or surgical footage.
Rohan Shad and his team demonstrated in Predicting post-operative right ventricular failure using video-based deep learning that pretraining on general human actions allows a model to better interpret the complex biological movements of a beating heart. Likewise, Isabel Funke showed that 3D CNNs initialized with action recognition weights could more accurately recognize surgical gestures in Using 3D Convolutional Neural Networks to Learn Spatiotemporal Features for Automatic Surgical Gesture Recognition in Video.
In all these cases, the underlying mechanism is the same: the "physics" of motion and the "logic" of 3D structure are universal enough that a model can learn them from YouTube and apply them to the operating room or the radiology suite.