In long, unedited video, the events a model must find occupy only a small slice of the timeline; most frames are background. Neural architecture search (NAS), the automated design of network structures, gets most of its speed from evaluating candidate designs cheaply: train them briefly on smaller inputs, or score them at initialization without training. For untrimmed video, the obvious way to make evaluation cheap is to make the input shorter. Shortening a video can mean trimming toward the action or keeping fewer frames. Both change the very property that makes untrimmed video hard. A short clip centered on an action has little background, so a candidate that ranks well on it may never have been tested on whether it can ignore distractors.
This note proposes cheap evaluation inputs that are shorter but keep the foreground fraction of real video, or lower it on purpose. They are built by thinning or repeating each video's own background segments. The proposal asks whether candidate rankings obtained on these inputs track the rankings that full training on full-length video would give, better than trimmed clips, plain short training, or existing training-free scores do. Put simply: the work tests whether the share of frames that contain the action is a property that cheap search must keep fixed, and builds a search procedure that keeps it fixed for almost no extra cost.
Core Claim
Let $\rho$ be the foreground fraction of a sequence, the share of its frames that fall inside annotated action instances. Let $L$ be the sequence length in feature steps. Cheap evaluation of a temporal architecture shrinks $L$, the number of training epochs, or both.
The claim has two parts. First, rank agreement with full evaluation is far more sensitive to changes in $\rho$ than to changes in $L$. This means a proxy can cut $L$ sharply as long as it holds $\rho$ at or below its natural value. Second, scoring each candidate by how slowly its accuracy falls as $\rho$ decreases adds ranking information that neither short training nor gradient-based initialization scores capture. This should matter most in the top tier of candidates, where existing training-free scores are known to fail.
The second part motivates a concrete proxy score for candidate $a$ trained for a small budget $b$:
$$S(a) = \frac{1}{|R|}\sum_{\rho \in R} \mathrm{mAP}_b(a; \rho, L_0)$$
Here $R$ is a small set of foreground fractions at and below the dataset's natural level. $L_0$ is a short fixed length. $\mathrm{mAP}_b$ is localization mean average precision after budget $b$, measured on thinned validation sequences.
This idea is a direct analogue of the sensitivity analysis in Rethinking Performance Estimation in Neural Architecture Search. That work found on images that epochs and depth cannot be compressed without breaking rankings, while image size can. The present proposal asks the same question for video length and foreground fraction, which no prior work treats as separate fidelity axes.
Research Program
The search space covers temporal localization heads that sit on frozen pre-extracted clip features. This follows the decoupled setup of NAS-TC, which searched 1D temporal convolution cells over frozen I3D features. The space is widened to include the background-handling designs that the sparsity literature found necessary. Those include dilated and separable temporal convolutions whose receptive fields can match action durations, as TAL-Net argued for its anchor towers. They also include separate action, background and context attention branches of the kind hand-designed in MAW-Net. Finally, they include a linear-time state-space mixing block like the one used for feature denoising in DRNet.
Restricting the space to heads has a practical purpose. It keeps full training of each candidate affordable, which is what makes ground truth possible. The plan is to train roughly two hundred sampled heads to convergence on THUMOS14 and on ActivityNet v1.3 features, with several seeds each. The result is a small tabular benchmark of true rankings for untrimmed temporal localization.
The sequence-construction procedure, called background thinning, works from each video's own annotations. It keeps every foreground segment and subsamples or repeats that video's background segments until the sequence reaches a target $(\rho, L)$. Two choices protect the realism of the background. Taking background only from the same video keeps the "informative background" that Annotation-Efficient Untrimmed Video Action Recognition showed cannot be treated as uniform clutter. The segments directly next to each action are always kept, because those near-boundary frames are where false boundaries arise.
With the benchmark and the thinning procedure in hand, three things get measured. The first is a sensitivity map: rank agreement as a function of $\rho$, $L$ and epochs, analyzed with the random-forest importance method of the budgeted-estimation work. The second is the proxy score $S$ above. The third is an initialization-time variant of $S$, which measures how strongly an untrained candidate's output separates foreground features from same-video background features at several values of $\rho$. That variant probes the open question in the file of whether a data-driven "actionness" signal, like the reconstruction-error ranking of Fast Temporal Activity Proposals, ranks architectures better than gradient-flow scores.
Each proxy then drives the same evolutionary search. The searches are compared on final mAP and on total GPU hours, counting proxy evaluation time.
Baselines
The naive reference is random search under the same GPU-hour budget, with candidates trained for the same short budget on full-length sequences. Alongside it, the best hand-designed head from the space is trained fully. TAL-Net is the hand-designed point of reference on THUMOS14, where it reports 42.8% mAP at tIoU 0.5 (TAL-Net). Our reimplementation on shared features will be reported next to that number rather than in place of it.
The strongest search method is NAS-TC's differentiable search, the method that relaxes discrete operation choices into weighted mixtures and optimizes them by gradient descent. It reports 39.3% mAP on Charades against 37.2% for the hand-designed Timeception, at 2.1 GPU days of search (NAS-TC). It will be run on our operation set, on the same frozen features and with the same seed count. Our searches will also be run on Charades to compare against that published figure directly. Beating it means equal or higher mAP at a fraction of its search cost, with the cost measured end to end.
The strongest cheap-evaluation baseline is plain budgeted estimation (Rethinking Performance Estimation), here meaning short training on full-length sequences with its fidelity settings tuned by the same sensitivity analysis. On images it cut search to 0.33 GPU days. No reported result was found on Lacuna for budgeted estimation on untrimmed video.
The training-free baselines are Synflow, NASWOT and the evolved EZNAS-A score (EZNAS). EZNAS-A reports a Kendall rank correlation of 0.65 on NASBench-201 CIFAR-10, against 0.57 for both Synflow and NASWOT. No reported rank correlation was found on Lacuna for any training-free score on untrimmed video.
The key ablations isolate the foreground fraction. One uses trimmed clips centered on actions, which have short $L$ and high $\rho$. Another uses uniform temporal subsampling, which has short $L$ and roughly natural $\rho$ but loses fine temporal resolution. The third is thinning with $R$ restricted to the natural $\rho$ alone, which tests whether sweeping $\rho$ downward adds anything.
All proxies get identical candidate pools and identical evolutionary-search settings, and their costs are charged in GPU hours. The proposal beats a baseline only if it improves Kendall correlation and top-10% recall against the full-training ranking, measured with a bootstrap over candidates, at equal or lower cost.
Why not just…?
Why not just run NAS-TC's differentiable search, since 2.1 GPU days is already affordable? Weight-sharing search keeps good local rank agreement but loses global agreement with converged performance, as Rethinking Performance Estimation documents on images. It is also prone to collapse toward parameter-free operations. Whether those failures get worse on noisy untrimmed sequences is untested. The ground-truth benchmark measures directly how good the architectures this search picks really are. If they match ours in quality, the main payoff left is cost, and the note will say so.
Why not just train each candidate for fewer epochs on the full videos? This is the real threat. If short training on full-length sequences already ranks well, thinning adds nothing beyond speed. The sensitivity map settles this. If epochs dominate and $\rho$ is unimportant, the core claim is false. Thinning can still be worth using if it gives equal ranks at much shorter $L$, because evaluation cost grows with sequence length. That is a smaller but honest result.
Why not just apply image zero-cost proxies to video heads? These scores were designed and validated on image convolution blocks. EZNAS reports that even on images they struggle to separate top-tier models (EZNAS). Nothing in them responds to whether a temporal receptive field fits sparse events. They are cheap enough to run on every candidate, so their correlation is reported rather than assumed bad.
Why not just trim around actions to make clips short? Trimmed inputs are where prior video search lives (Video Action Recognition with Neural Architecture Search, Video Action Recognition Via Neural Architecture Searching). Yet on untrimmed data, picking 25 frames with a learned policy beat using all frames, 83.81% against 82.53% mAP on ActivityNet v1.3 (MARL frame sampling). That suggests background content shapes which models do well. The trimmed-clip ablation tests exactly this point.
Proposed Experiments
The first decisive check runs on THUMOS14. It compares Kendall correlation and top-10% recall of the thinned proxy against trimmed clips, uniform subsampling and short full-length training, at matched GPU hours. The prediction is that trimming loses the most rank agreement, and that the loss concentrates in candidates with explicit background branches.
The second check uses ActivityNet v1.3 as a contrast case. If the claim is right, the advantage of preserving the foreground fraction should shrink on the dataset where actions fill more of the video. A similar gap on both datasets would point to some cause other than sparsity.
The third check uses QVHighlights, where ground-truth moments cover less than 30% of each video (DRNet). It tests whether a ranking found under one value of $\rho$ transfers to a text-queried task at another.
The final check runs full searches driven by each proxy. They are compared against differentiable search on THUMOS14 mAP at tIoU 0.5 and on Charades mAP, with total search GPU hours reported beside the cost of running the discovered head.
The result that would kill the idea is any of the following: rank agreement that does not depend on $\rho$; the thinned proxy failing to beat short full-length training; or the initialization-time variant doing no better than Synflow.
References
- Rethinking Performance Estimation in Neural Architecture Search
- NAS-TC: Neural Architecture Search on Temporal Convolutions for Complex Action Recognition
- Rethinking the Faster R-CNN Architecture for Temporal Action Localization
- Weakly-supervised temporal action localization using multi-branch attention weighting
- Denoise-then-Retrieve: Text-Conditioned Video Denoising for Video Moment Retrieval
- Annotation-Efficient Untrimmed Video Action Recognition
- Fast Temporal Activity Proposals for Efficient Detection of Human Actions in Untrimmed Videos
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Video Action Recognition with Neural Architecture Search
- Video Action Recognition Via Neural Architecture Searching
- Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition