Anchoring Video-Pretrained 3D Networks Across Imaging Modalities

A 3D scan and a short video clip have the same shape. Both are stacks of 2D frames, and in both the content changes smoothly from one frame to the next. Earlier work used this fact to start 3D medical networks from weights learned on natural video, and this helps when labelled scans are scarce. The evidence so far covers only CT and only whole-volume classification. We propose to test whether that video-learned knowledge of "how structures move between frames" carries over when the scanner itself changes, from CT to MRI. Our working hypothesis is that it is the part of the network that should survive. Outline continuity between slices depends on anatomy, not on whether the pixel values are X-ray attenuation or magnetic relaxation.

The concrete proposal has three parts. We train a video-initialized 3D segmentation network on a small labelled CT set. We attach extra prediction heads to its intermediate, coarse layers, a technique called deep supervision. These heads are forced to output low-resolution organ masks, which describe shape and position and say little about image appearance. When we then adapt to a handful of MRI scans, the coarse, depth-direction part of the network stays fixed and acts as an anchor. Only the in-plane, appearance-sensitive layers and the fine output head relearn. If the hypothesis is right, this beats ordinary fine-tuning in the 1-to-8-scan regime. If it is wrong, a layer-by-layer probing study will show where modality information actually lives in these networks. That result is useful in its own right.

Core Claim

In a 3D network initialized from natural video, the information that transfers across imaging modalities sits mostly in two places: the through-plane (depth-axis) operators and the coarse spatial scales. Appearance-specific information sits mostly in the in-plane operators of the early layers. Coarse deep supervision on the source modality makes this split sharper, because the intermediate features are tied to modality-free shape targets. Freezing the anchored part and adapting only the appearance part should then give better few-shot cross-modality segmentation than three alternatives:

  • end-to-end fine-tuning of the same video-pretrained network with strong intensity augmentation;
  • deep supervision without video initialization;
  • video initialization without coarse anchoring.

What Prior Work Establishes and Leaves Open

Video pretraining helps in small samples, but only in CT and mostly for classification. Two papers motivate the core mechanism. AppendiXNet fine-tuned a Kinetics-pretrained 3D ResNet-18 on 438 abdominal CTs and raised test AUC from 0.724 (from scratch) to 0.810. Video pretraining advances 3D deep learning on chest CT tasks showed similar gains across seven 3D architectures: mean AUC on RSNA pulmonary embolism rose from 0.573 to 0.719. That paper also found that Kinetics-400 pretraining beat in-domain pretraining on 1,837 RadFusion CTs. Both studies supervise only the final output, stay within CT, and predict at the level of the whole volume. We move to dense, voxel-level prediction across modalities, and we ask which parts of the video network transfer.

Video features may be too coarse for fine boundaries. One paper constrains our design. Inter-slice super-resolution of MRI pretrained on video frame interpolation but still needed an in-domain stage on 350 knee MRIs plus per-subject self-supervised tuning to reach 30.88 dB PSNR. This suggests video priors alone do not recover fine boundaries. We read it as support for a split design: video knowledge anchors the coarse levels, and the fine, appearance-dependent levels must be relearned. We test that reading directly rather than assume it.

Intermediate supervision helps in few-shot and shortcut settings, but has not been tested for modality shortcuts. Two papers supply the method ingredient and the evidence that intermediate supervision matters.

  • Anatomical auxiliary supervision with dual contrastive learning adds superpixel pseudo-label supervision on slices that do not contain the target organ. It reaches 76.36% Dice in 1-shot segmentation on CHAOS-T2 MRI. Removing its contrastive and auxiliary parts drops this to 62.07%. That work processes 2D slices from an ImageNet backbone and trains each modality separately.
  • Targeted Activation Penalties shows that penalizing intermediate activations stops shortcut cues that re-emerge in deeper layers when only input gradients are penalized. That work is restricted to 2D images within a single modality.

We treat source-modality intensity statistics as exactly such a shortcut. We ask whether coarse intermediate heads in a 3D video network suppress it.

The cross-modality gap is real, and existing remedies use no video prior. The CHAOS challenge found that models handling CT and MR jointly fell from about 0.97 to 0.88 Dice. Even strong models kept surface errors around 21 mm, so overlap scores can hide bad boundaries. Meta-hallucinator tackles MR-to-CT cardiac transfer with meta-learned synthetic images and gains 6.7% Dice over MT-UDA in the 4-shot setting. Self-Sampling Meta SAM adapts a frozen 2D foundation model with adapters and meta-learning, but evaluates CT and MRI separately. None of these uses a 3D continuity prior. Our approach changes no images and does no meta-learning; it only chooses what to freeze and where to supervise. Image-translation methods could therefore be stacked on top of it.

Research Program

Backbone. We use a video backbone whose spatial and temporal operators are separate parameters, so "in-plane" and "depth-axis" weights can be frozen independently. Candidates are a factorized-convolution 3D ResNet and a 3D Swin-T, both pretrained on Kinetics-400 following the recipe of the chest CT study. Video time maps to slice index. The three RGB channels are collapsed by averaging the first-layer weights for single-channel input. A lightweight 3D decoder with skip connections is added on top.

Supervision. Each decoder stage $s$ gets a head that predicts the organ mask downsampled to that stage's resolution. The source-training loss is

$$\mathcal{L} = \sum_{s=1}^{S} w_s \, \ell\big(\hat{y}_s, \mathrm{down}_s(y)\big),$$

where $\hat{y}_s$ is the head-$s$ prediction, $\mathrm{down}_s(y)$ is the ground-truth mask $y$ downsampled to stage $s$, $\ell$ is Dice plus cross-entropy, and $w_s$ are stage weights. The weights are fixed by a single rule, halving per finer stage, so that weight tuning does not become a hidden advantage.

Adaptation to MRI. With $k \in \{1, 4, 8\}$ labelled MRI volumes, we freeze three components: the depth-axis operators throughout, the deepest encoder blocks, and the coarse heads. We train the in-plane operators of the early blocks and the finest head. The frozen coarse heads also supply a consistency target on the unlabelled MRI volumes: the fine prediction, downsampled, must agree with them.

Slice-spacing matching. Abdominal MRI slices are often several millimetres thick, while video frames are evenly spaced. As a controlled factor, we resample the through-plane spacing so that organ outlines move between adjacent slices by an amount comparable to typical motion between adjacent Kinetics frames. We then sweep this spacing. This tests whether the anisotropy problem noted for video transfer is partly a "frame-rate mismatch."

Diagnostic study. For every layer and each axis-group of parameters, we fit two linear probes:

  • a modality probe that predicts CT versus MR from pooled features (lower accuracy means more modality-invariant features);
  • an organ probe trained on CT features and tested on MR features (higher means more transferable features).

We compare these probe curves across video, ImageNet-inflated, scratch, and in-domain-pretrained initializations, each with and without deep supervision. The core claim predicts a specific pattern. Video-initialized, deeply supervised networks should show their lowest modality-probe accuracy and highest cross-modality organ-probe accuracy in the depth-axis operators and coarse stages. The other initializations should not show this pattern.

Baselines

Do-nothing reference. This is a 3D U-Net trained from scratch, with the same multi-scale deep supervision used by most modern medical segmenters. It is evaluated two ways: on the source CT only and applied directly to MRI, and also fine-tuned end-to-end on the $k$ MRI scans. It shows how much of the gap remains without any prior. The CHAOS result, a drop to 0.88 Dice under joint CT-MR training, is the closest published reference. It is not the same protocol, so we rerun it ourselves.

Strongest direct competitor. The same Kinetics-pretrained backbone and decoder, fine-tuned end-to-end on CT and then on the $k$ MRI scans. It uses aggressive intensity augmentation: random intensity remapping, contrast inversion, and histogram randomization. This is the natural extension of Ke et al. and AppendiXNet to segmentation. It is the most likely way the proposal fails, because augmentation alone may already wash out intensity shortcuts. No reported result was found on Lacuna for video-pretrained 3D networks on cross-modality segmentation, so this baseline exists only as our own run. All arms receive identical augmentation, data splits, epochs, and a fixed budget of twenty hyperparameter trials each.

Published few-shot references. On MM-WHS, Meta-hallucinator is the strongest published cross-modality method. The file reports only a relative gain: 6.7% Dice and 0.9 mm average surface distance over MT-UDA with 4 source shots. No absolute Dice for Meta-hallucinator was found on Lacuna. We will therefore rerun it under its own protocol, with a matched number of labelled scans, rather than quote a number. On CHAOS-T2, AAS-DCL reports 76.36% mean Dice in 1-shot within-MRI segmentation. This is a within-modality reference, a target the cross-modality setting should approach. It is not a like-for-like competitor. SSM-SAM reports only relative gains on ABD-30 and ABD-MRI (10.21% and 1.80% Dice over prior few-shot methods). We include it as the foundation-model alternative, run with its released settings on our splits.

Ablations isolating the key ingredient. Four ablations separate the effects:

  • Anchoring without video: coarse-anchored adaptation applied to scratch and ImageNet-inflated initializations.
  • Video without anchoring: deep supervision during source training, then full fine-tuning.
  • Axis versus depth freezing: freezing by depth only (the usual "freeze early or late layers") instead of by axis plus scale.
  • Consistency term: removing the consistency term on unlabelled MRI.

Success criterion. Beating the strongest competitor means higher mean Dice and lower average symmetric surface distance on MRI at $k=1$ and $k=4$. The paired difference across five random source/target splits must have a 95% interval that excludes zero. We expect any advantage to shrink at $k=8$, and we report that trend.

Why not just…?

Why not just fine-tune the video-pretrained network with heavy intensity augmentation? This is the real threat. Nothing in the file shows augmentation fails here; the file contains no cross-modality video-transfer experiment at all. We make it the primary baseline and give it the same tuning budget. If it matches the anchored method, the method claim fails, and the paper becomes the diagnostic study. Even then, the probes tell practitioners which layers carry modality information, and whether augmentation removes that information or merely hides it.

Why not just use standard deep supervision with a scratch or ImageNet start, as most medical segmenters already do? Deep supervision alone is already in the do-nothing baseline. The question is whether the video prior adds anything for dense, cross-modality prediction. The super-resolution study gives a reason to doubt this, since video priors alone did not recover fine MRI detail. The "anchoring without video" ablation settles the question. If anchoring helps scratch and ImageNet starts equally, the video component is unnecessary and we will say so.

Why not just pretrain on CT and MRI archives instead of videos? Ke et al. found Kinetics-400 beat 1,837 in-domain CTs, but only for CT classification. In-domain pretraining that mixes both modalities could plausibly learn invariance directly. We include a self-supervised in-domain initialization of matched compute in the probe study and the main comparison. We expect video to win at this archive scale, but we treat that as an open question.

Why not just translate MRI into CT-like images, or use a 2D foundation model? Meta-hallucinator and SSM-SAM are strong in their settings. However, translation adds a generator that is itself hard to train from very few scans. SAM-style models process slices in 2D and carry a large adaptation cost. Our method is orthogonal to translation, and we report its combination with hallucination on MM-WHS. Falling behind Meta-hallucinator on cardiac data while winning on abdomen would point to a domain-specific limit, which we would report.

Proposed Experiments

Abdominal benchmark. The source is ABD-30 CT (30 labelled scans: liver, spleen, both kidneys). The target is CHAOS MRI, using T2-SPIR and, separately, T1-DUAL, with $k \in \{0, 1, 4, 8\}$ labelled volumes. Evaluation is on held-out CHAOS MRI patients, restricted to organs labelled in both sources. Following the CHAOS observation that overlap hides boundary failure, we report Dice, average symmetric surface distance, and maximum symmetric surface distance.

Cardiac benchmark. We use the reverse direction, MR to CT, on MM-WHS 2017 under the Meta-hallucinator 1- and 4-shot protocols. This checks that the effect is not tied to one anatomy or one transfer direction.

Three decisive checks.

  1. Does anchored adaptation beat augmented end-to-end fine-tuning of the same video network at $k \le 4$, by the interval criterion above?
  2. Do the layer and axis probes show modality invariance concentrated in depth-axis and coarse-scale parameters, only for video-initialized and deeply supervised networks?
  3. Does the slice-spacing sweep produce a peak near the "matched frame-rate" spacing, rather than a flat curve?

If the first check fails but the second holds, the video prior is invariant but the freezing scheme is the wrong way to use it. If the second check fails, the continuity-transfer story itself is wrong for dense cross-modality tasks. That would sharpen the tension between the chest CT and super-resolution findings.