Pre-training a video model is expensive. Tuning its settings (learning rate, weight decay, how much of each clip to hide) is worse, because every candidate setting normally costs another full run. We propose to hide the tuning cost inside a cheap part of training that already exists. Recent video pre-training methods grow the clip length during training, starting on 2-frame clips and ending on 16-frame clips. We treat those short-clip stages as a tournament. Many candidate settings train in parallel on short clips, the weak ones are dropped at each point where clip length increases, and the survivor keeps its weights and simply becomes the final model. Most of the search compute is therefore not wasted: the winning branch is the production run.
This only works if a setting that looks good on 2-frame or 4-frame clips is still good on 16-frame clips. For spatial resolution in image models this is known to hold. For clip length in video models nobody has checked. We expect it to hold for optimizer settings but to fail for the masking ratio, because the argument for hiding 90–95% of a video is temporal redundancy, and a 2-frame clip has little of it. The project first measures which hyperparameters keep their ranking as clips get longer. It then builds a search procedure that commits early on the settings whose ranking holds and postpones the settings whose ranking does not.
Core Claim
A masked autoencoder learns by reconstructing hidden patches of its input. For masked video autoencoders trained with a progressive frame curriculum, the curriculum's early stages are a nearly free fidelity ladder for hyperparameter search. A search that halves its candidates at curriculum boundaries, and reuses the surviving weights, should reach the accuracy of a separately tuned model at close to the cost of one untuned run. It should do so only if the search is told which hyperparameters transfer across clip length.
The claim is falsifiable at two points. The first is the rank correlation of configurations between short-clip and full-clip training. The second is whether the combined search-plus-training cost undercuts tuning with standard epoch-based early stopping at equal final accuracy.
Where This Sits
The mechanism we build on is Efficient VideoMAE via Temporal Progressive Training (TPT). It pre-trains a video masked autoencoder on 2→4→8→16-frame clips, splitting the budget roughly 12.5/25/50/12.5%. This matches standard VideoMAE at 2x lower cost on Kinetics-400 and 3x lower cost on Something-Something V2, the latter a motion-centric benchmark where actions depend on fine temporal order. TPT also motivates the proposal: its authors report that choosing the curriculum and the per-stage learning-rate schedules required an expensive preliminary search. That is exactly the overhead we want to remove.
The model itself is VideoMAE, which hides 90–95% of space-time patches using "tube" masks, meaning the same spatial mask is repeated across all frames. It justifies this ratio by temporal redundancy. That justification is what makes us suspect the ratio does not transfer down to 2-frame clips.
Three HPO papers supply methods and constrain the design:
- BOHB combines Bayesian proposal of candidates with successive halving, where many candidates get a small budget and only the best are promoted. Its fidelity is training time. We keep the halving logic but change what a cheap evaluation means.
- Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks showed that for 2D CNNs the importance and optimal values of learning rate, batch size and weight decay are invariant to image resolution. It cut search time by 9–42%, but it left temporal fidelity unstudied.
- Multi-level Training and Bayesian Optimization for Economical Hyperparameter Optimization models full-training error as a function of early-training error. It uses only epoch-based fidelity and small image benchmarks.
Efficient Hyperparameter Tuning via Trajectory Invariance Principle found that in language transformers the loss trajectory depends mainly on the product of learning rate and weight decay. Holding that product fixed collapses a 2D search into a 1D one. We test whether this still holds across frame-count changes. TPT's need for stage-specific schedules is a warning that it may not.
Low-Fidelity Video Encoder Optimization for Temporal Action Localization (LoFi) reduces spatial and temporal fidelity so that end-to-end fine-tuning fits on one GPU. It uses those reductions as fixed training proxies, not as a search tool, and notes that its own downscaling ratios needed manual tuning.
CARBS does cost-aware Bayesian search along a performance-cost frontier. It has been shown on language, RL and Imagenette, not video.
None of these uses the training curriculum itself as the fidelity axis, and none reuses the search's weights as the final model.
Research Program
Stage 1: an invariance study. We will sample a modest set of configurations (on the order of 24–32) over peak learning rate, weight decay, masking ratio, warmup length, and the per-stage learning-rate restart used by TPT. The model is a ViT-Small or ViT-Base VideoMAE, pre-trained on Something-Something V2 and on a fixed Kinetics-400 subset.
Each configuration is run independently at 2, 4, 8 and 16 frames on a matched short schedule. Each is then scored by fine-tuned top-1 accuracy, together with the cheaper reconstruction loss and a linear-probe accuracy. For each hyperparameter we measure how well rankings at $T$ frames predict rankings at 16 frames, using Spearman correlation computed on marginal slices of that hyperparameter. We also check whether the learning-rate-times-weight-decay product still collapses the loss curves across frame counts.
The expected output is a per-hyperparameter "transfer horizon": the shortest clip length whose ranking is trustworthy. This is the scientific result of the project whether or not the search method wins.
Stage 2: the search procedure. Candidates are launched on 2-frame clips. At each clip-length boundary the bottom fraction is dropped on the validation proxy, and survivors continue from their own weights. The procedure uses the transfer horizons from Stage 1 in two ways. A hyperparameter with a short horizon is decided early, and its losers are pruned. A hyperparameter with a long horizon, likely the masking ratio, is either held at a shared value until the rung where its ranking becomes reliable, or split into sibling branches at that rung from a common parent checkpoint.
New candidates can be proposed with BOHB's density model. The final survivor finishes the 16-frame stage and is the delivered model.
The accounting is simple. Let $C_{\text{run}}$ be the cost of one TPT run and $n_k$ the number of branches alive in stage $k$, which takes budget fraction $b_k$. The total cost is
$$C_{\text{search}} = C_{\text{run}} \sum_k n_k b_k.$$
The final branch's cost is included in this sum, so the overhead is only the pruned branches. Because the early stages are both the cheapest per step and the most heavily populated, a population of 16 halved at each boundary costs a small multiple of one run rather than 16 runs.
Stage 3: downstream transfer. We check whether the selected pre-training settings also produce better backbones for HMDB-51 and ActivityNet-v1.3 action localization. Pre-training accuracy and transfer can disagree; LoFi documents that mismatch between clip classification and localization.
Baselines
Naive reference: the published default recipe. This is the VideoMAE recipe with TPT's published curriculum, trained once at our model scale. It sets the zero-search-cost accuracy. For scale, VideoMAE reports 87.4% top-1 on Kinetics-400 and 75.4% on Something-Something V2 at large scale (VideoMAE). TPT reports +0.4–0.5% on K400 and +0.2–0.6% on SSV2 over VideoMAE at equal compute (Efficient VideoMAE via Temporal Progressive Training). We will rerun the recipe at our smaller scale rather than compare to those numbers directly. Any search method must beat this default by more than seed noise, which we will estimate from three seeds.
Strongest general competitor: BOHB with epoch-based fidelity. This is BOHB run on full 16-frame clips, with fidelity defined as a fraction of training epochs and each promoted configuration restarted or resumed as BOHB normally does. It is the method most practitioners would reach for. On video its reported results do not exist: no reported result found on Lacuna for BOHB or any multi-fidelity HPO on video pre-training. Its closest figure, 2.78% CIFAR-10 test error in 33 GPU-days, is not comparable.
It gets the same search space, the same total GPU-hour budget, and the same final evaluation. Beating it means reaching equal final fine-tuned accuracy at clearly lower total cost, or higher accuracy at equal cost, across both datasets.
Spatial-fidelity competitor: spatial downscaling. This is the Increasing Image Size approach of Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks, extended to video: halving uses 16-frame clips at reduced resolution. It isolates whether the temporal axis matters for search, as opposed to any cheap axis.
Ablations that isolate the key ingredients. We run curriculum-rung halving with the invariance schedule removed, so all hyperparameters are pruned at every rung. This tests whether knowing the transfer horizons matters. We also run curriculum-rung halving where the winner is retrained from scratch instead of continued. This tests how much of the saving comes from weight reuse, as opposed to cheap short-clip evaluation.
Why not just…?
Why not just use BOHB with epoch-based early stopping? Epoch-based rungs still pay full 16-frame cost on every step, while TPT shows most of that cost is avoidable during pre-training. Early epochs on long clips may also mostly measure how quickly the model learns spatial reconstruction, a concern raised by VideoMAE's redundancy argument. That is a hypothesis, not an established fact, and it is a real threat: if early-epoch rankings on 16-frame clips are simply more faithful than 2-frame rankings, BOHB could win at modest cost. The equal-budget comparison and the Stage 1 correlation table settle this directly.
Why not just tune the product of learning rate and weight decay with a 1D sweep, as trajectory invariance suggests? If trajectory invariance holds for video autoencoders across frame counts, this is a strong and cheap alternative for the optimizer settings. We will adopt it inside our search rather than compete with it. It does not address the masking ratio or the per-stage schedule, which TPT reports needing to tune. Stage 1 tests whether the collapse survives clip-length changes. If it does, and the masking ratio also transfers, the contribution shrinks to "sweep one number on 2-frame clips," and we will report that plainly.
Why not just tune on 2-frame clips and transfer everything, with no halving? This is the degenerate version of our method, with a transfer horizon of 2 frames for every hyperparameter. It is exactly what the invariance study evaluates. Our prediction that the masking ratio's optimum shifts with clip length is the reason we expect it to fail. If the prediction is wrong, this simpler scheme should replace ours.
Why not just spatially downscale, since that is already known to preserve rankings? Spatial invariance was shown for 2D CNNs only. TPT identifies clip length as the main cost driver of video transformers, and LoFi found combined spatial-temporal cycling better than either axis alone for localization. The spatial baseline above will measure the difference, and a hybrid rung (short clips at low resolution) is a natural extension if both axes help.
Decisive Checks
The first check is the Stage 1 table. If optimizer settings keep Spearman correlation above roughly 0.7 from 4 frames onward, and the masking ratio does not until 8 frames, the design is justified. If nothing transfers below 8 frames, the savings disappear and the project reduces to a negative result about temporal fidelity. That result is still worth publishing, since it bounds how far multi-fidelity search can go for video.
The second check is the cost-accuracy curve. We plot final fine-tuned top-1 on SSV2 and Kinetics-400 against total GPU-hours, including search, for all baselines at three budgets. The proposal succeeds if its curve lies strictly above epoch-based BOHB and spatial downscaling, and if the no-reuse ablation shows that weight reuse accounts for a substantial share of the gap.
The third check is transfer to HMDB-51 and ActivityNet-v1.3. Settings chosen by the search should not transfer worse than the defaults. For localization, the reference point is G-TAD with LoFi at 34.49 average mAP versus 33.59 without it (Low-Fidelity Video Encoder Optimization for Temporal Action Localization). We compare against our own rerun at matched backbone scale.
References
- Efficient VideoMAE via Temporal Progressive Training
- VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
- BOHB: Robust and Efficient Hyperparameter Optimization at Scale
- Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks
- Multi-level Training and Bayesian Optimization for Economical Hyperparameter Optimization
- Efficient Hyperparameter Tuning via Trajectory Invariance Principle
- Low-Fidelity Video Encoder Optimization for Temporal Action Localization
- Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training