Depth-Assigned Targets for Masked Video Autoencoders Trained on Smoothly Augmented Clips

A masked video autoencoder learns from unlabeled video by hiding most of each clip and training a network to fill in the missing pixels. Current systems of this kind use almost no data augmentation. Separately, video recognition work has found that augmentations which drift smoothly over time, such as slowly changing brightness or a gradual zoom, make models more robust than augmentations held fixed across a clip. Adding such drift to a masked autoencoder raises a question nobody has answered: should the network reconstruct the drifting pixels it was shown, or the clean pixels underneath? Our proposal is that the answer should depend on depth. We generate the drift ourselves, so we know its exact parameters, and that gives us both targets for free. We ask shallow layers to reconstruct what was observed and deep layers to reconstruct the undrifted clip. The hierarchy then learns to represent the nuisance first and to discard it higher up.

This matters because two properties are currently in tension. Pixel reconstruction keeps fine appearance detail, which is its strength, but it gives the representation no pressure to ignore lighting changes or camera zoom. Contrastive methods add that pressure, but they do it with a separate teacher network and a global clip-level loss. Using known augmentation parameters as supervision could deliver nuisance invariance inside a plain reconstruction objective. It would need only a few single-block decoders and no second network.

Core Claim

In masked video modeling, a temporally smooth augmentation with known parameters defines two reconstruction targets for each masked location. The first is the observed, augmented clip $x$. The second is the canonical clip $\bar{x}$, which is the same crop with the time-varying part of the augmentation removed. We attach lightweight decoders $h_\ell$ to several intermediate depths $\ell$ of a flat Vision Transformer, the standard patch-based transformer (ViT). We then train with

$$\mathcal{L} = \sum_{\ell} \lambda_\ell \,\big\| M \odot \big(h_\ell(z_\ell) - y_\ell\big) \big\|^2, \qquad y_\ell = x \text{ for shallow } \ell,\; y_\ell = \bar{x} \text{ for deep } \ell,$$

where $z_\ell$ is the encoder's token output at depth $\ell$, $M$ is the binary mask over hidden spacetime patches, and $\lambda_\ell$ are fixed weights. The claim is that this assignment works better than reconstructing $x$ everywhere, better than reconstructing $\bar{x}$ only at the final layer, and better than the reversed assignment. The expected gains are in frozen-feature quality and in robustness to test-time drift, while fine-tuned accuracy on motion-centric benchmarks stays at least equal.

We add one secondary mechanism. Tube masking hides the same screen location in every frame. Under a synthetic zoom, content slides out from under the tube into visible patches, and the redundancy that tube masking was meant to remove comes back. Because the warp is known, we can instead warp the mask along with the content, so that the same underlying content stays hidden across frames. We call these content-anchored tubes. This needs no point tracker.

Relation to Prior Work

The masking side of the method comes from VideoMAE. That paper introduced tube masking at 90–95% masking ratios on a flat ViT with only crop and flip augmentation. VideoMAE V2 scaled the approach with decoder-side masking. Efficient VideoMAE via Temporal Progressive Training cut its cost with a curriculum over clip length. None of the three changes the augmentation or the reconstruction target. We keep their encoder and masking, and change only what is reconstructed, at which depth, and how the mask moves.

The augmentation comes from DynaAugment. DynaAugment samples augmentation strength as a random sum of sinusoids over time. It reported gains of 0.5–1.7% on Kinetics-400 top-1 accuracy, and it found that independent per-frame augmentation does worse than static augmentation. That paper motivates this proposal. It also constrains it: independent per-frame jitter is ruled out, and DynaAugment itself warns that its interaction with regularizers is delicate to tune. It was evaluated only in supervised training. CVRL established clip-consistent augmentation for contrastive video learning. Our canonical target $\bar{x}$ is exactly a CVRL-style consistent clip.

The multi-depth supervision comes from LocalMIM. LocalMIM attaches single-block decoders to intermediate layers and supervises them fine-to-coarse on static images, reporting 3.1× less computation than MAE on ViT-B. It varies the scale of the target (pixels or HOG edge descriptors below, coarse targets above). We vary a different axis, nuisance-bearing versus nuisance-free, and we move the idea to video.

ConvMAE shows that hierarchical encoders need block-wise masking to avoid convolutional leakage and partially masked tokens. That problem is unsolved in 3D, and it is the reason we stay with a flat ViT. CMAE-V is the closest competitor in intent. It obtains invariance by adding a momentum contrastive branch with temporal-shift views to video reconstruction. We aim for similar invariance without a second network. MoSiC enforces temporal consistency by propagating clusters along tracker-estimated trajectories, and it acknowledges tracker errors. Our correspondences are exact because we synthesize the motion ourselves.

Research Program

We build on a ViT-B VideoMAE with 16-frame clips. During pretraining, each clip receives a base augmentation that is constant across the clip (crop, flip, color). On top of that we add DynaAugment-style smooth drift in two families. The photometric family varies brightness, contrast and hue over time. The geometric family varies scale, as a gradual zoom about a sampled center. The canonical clip $\bar{x}$ is the base augmentation alone. It costs one extra render per clip and no extra encoder pass.

Decoders are single transformer blocks, following LocalMIM. We place them after blocks 4, 8 and 12. The default assignment gives the observed target to block 4 and the canonical target to blocks 8 and 12. Block 8 is included in the canonical group so that we can study where the switch point should fall. Content-anchored masking works as follows. We sample the tube mask in the base-frame coordinate system and map it to each frame through that frame's known affine transform. A patch is marked hidden if most of its area is covered, and we rebalance so the per-frame masking ratio stays at the target.

The analysis has three parts. First, we measure how much zoom and lighting information each layer carries by linearly probing the drift parameters from $z_\ell$. We expect drift to be decodable from shallow layers and much weaker in deep layers. Second, we measure leakage directly. We report the reconstruction error of a trivial "copy the nearest visible patch from an adjacent frame" predictor under screen-anchored versus content-anchored tubes. Third, we sweep the depth at which the target switches from observed to canonical.

Baselines

The naive reference point is VideoMAE exactly as published: static crop and flip, screen-anchored tubes, a single final decoder on the observed clip. It is retrained under our budget. The strongest external number in the file for this setting is from CMAE-V at ViT-B scale with 1600 epochs. It reports 80.7% Kinetics-400 top-1 for VideoMAE and 82.2% for CMAE-V with ConvViT-B, and 70.3% versus 71.6% on Something-Something V2. VideoMAE also reports 62.6% on HMDB-51 when pretrained on HMDB-51 alone. The file does not state the backbone behind VideoMAE's 87.4% and 75.4% figures, so we do not treat them as ViT-B targets.

The first strong internal baseline adds DynaAugment drift to VideoMAE and reconstructs the observed clip at the final layer only. This is the obvious combination. Nobody has reported it (no reported result found on Lacuna for DynaAugment inside masked video pretraining), so we will produce that number ourselves.

The strongest competitor on the invariance question is CMAE-V. We run it with the same ViT-B, the same drift augmentations applied to its views, the same epoch count and the same learning-rate sweep. We charge it for its extra teacher forward pass, reporting accuracy both at equal epochs and at equal GPU-hours.

We also run TPT at equal budget. TPT reports gains of 0.4–0.5% on Kinetics-400 and 0.2–0.6% on Something-Something V2 over VideoMAE at matched compute. That sets the size of gain that counts as meaningful for a cheap change.

The ablations isolate the key ingredient. They are: the canonical target at the final layer only; the observed target at all heads, which is LocalMIM moved to video; the reversed assignment; and screen-anchored versus content-anchored tubes with depth-assigned targets held fixed. All runs share data, 800 pretraining epochs, and a three-point learning-rate search on a held-out split. We would count the method as beating these baselines if it improves linear-probe top-1 over the drift-augmented observed-only baseline by more than the seed spread (three seeds) on both Kinetics-400 and Something-Something V2. It must also match or exceed CMAE-V's fine-tuned accuracy at equal GPU-hours, and it must lose no fine-tuned accuracy on Something-Something V2.

Why not just…?

Why not just add DynaAugment to VideoMAE and keep the pixel loss? This is the most likely threat. If it matches our method, the depth assignment adds nothing and we will say so. Our concern is specific. With a single final target, the drift is part of what must be predicted. The deepest layer is then pushed to encode lighting and zoom, which is the opposite of invariance. The layerwise probes of drift parameters will show which regime actually occurs.

Why not just reconstruct the canonical clip at the final layer? This is simpler and needs no extra heads. Its weakness is that the whole network is asked to undo the drift, with no layer allowed to represent it first. LocalMIM's finding that shallow layers learn slowly under a single distant loss suggests this will train worse. The final-layer-canonical ablation tests exactly this, and if it ties our method we will adopt it as the simpler recommendation.

Why not just use a contrastive branch, as CMAE-V does? CMAE-V already shows that contrast adds global invariance, with 82.2% versus 80.7% on Kinetics-400. It achieves this with a momentum teacher and 1600 epochs. Our bet is that known augmentation parameters give similar invariance for the cost of three single-block decoders. The equal-GPU-hours comparison decides this. If CMAE-V wins at equal compute, the proposal fails on its main practical argument.

Why not just use a hierarchical encoder such as Video Swin with ConvMAE-style masking? Video Swin Transformer gives native multi-scale features, but it was trained with supervision. ConvMAE documents that downsampling under random masks produces partially masked tokens and convolutional leakage, and a 3D fix does not yet exist. Taking that route would mix an unsolved architecture problem into the target question. We keep a flat encoder so that any gain can be attributed to the targets alone.

Proposed Experiments and Decisive Checks

Pretraining uses Something-Something V2 and Kinetics-400 separately at ViT-B. A small-data run pretrains on HMDB-51 alone, where VideoMAE's 62.6% is the reference. Evaluation covers fine-tuned top-1 accuracy, linear-probe top-1 on frozen features, and transfer to UCF-101 and HMDB-51.

We also test robustness to drift. We apply held-out smooth lighting and zoom schedules, with frequencies outside the training range, to the Kinetics-400 and Something-Something V2 validation sets, and report the accuracy drop. This is a constructed test, not an existing benchmark. As a dense check, we propagate labels with frozen features on DAVIS, following MoSiC's evaluation.

The first decisive check is whether the drift-augmented observed-only baseline beats plain VideoMAE at all. If it does not, drift and masking conflict and our targets are the fix to be tested. The second is whether depth-assigned targets beat final-layer-canonical on linear probing while probes show drift information decaying with depth. The third is whether content-anchored tubes reduce the copy-predictor's error advantage under zoom and improve Something-Something V2 accuracy. One risk must be checked: some Something-Something V2 classes involve camera-like zoom. If geometric invariance hurts those classes, we restrict canonicalization to photometric drift and report the split by class.