Camera-based gesture recognizers work well in the lab and fail in kitchens, cars and workshops. The usual causes are a light that ramps up, a shadow that sweeps across the hand, a skin-colored object in the background, or a head-mounted camera that shakes. These failures have one thing in common: they unfold over time. Three research communities each hold part of the answer, and none has the whole. Gesture researchers add depth sensors or distill from them. Architecture-search researchers have built cheap "zero-cost proxies," scores computed on an untrained network in milliseconds that try to predict how a trained version will behave, including how robust it will be. Benchmarking researchers keep finding that claimed gains disappear once training protocols are held fixed. Every robustness proxy so far has been built and checked on single still images. No one knows whether a small streaming video network's robustness to time-varying interference is set by its wiring at all, as opposed to its training recipe.
We propose to settle this with a frozen-protocol benchmark. A few hundred small streaming gesture architectures will each be trained under one fixed recipe on EgoGesture and then tested against temporally structured corruptions. On that ground truth we test one specific hypothesis. Robustness to coherent temporal interference depends mainly on where and how a network mixes information across frames. It can therefore be predicted before training, but only by a proxy that perturbs the input along time. Per-frame image proxies cannot capture it, and neither can a ranking by clean accuracy.
Core Claim
Two claims are at stake, and the benchmark is designed so that both can fail.
The first is that architecture matters for temporal robustness once the protocol is fixed. Among networks of similar compute, trained with the same augmentation, splits and number of epochs, accuracy under corruptions such as illumination drift or camera shake should differ by more than seed noise. It should also differ more than clean accuracy alone would predict. Neural Architecture Design and Robustness: A Dataset found exactly this kind of spread for still images: at identical parameter count, adversarial robustness ranged from 20% to 41% across NAS-Bench-201. Whether the effect survives in video, and under natural rather than adversarial interference, is open.
The second is that the useful signal at initialization comes from structured temporal perturbations. A slow brightness ramp, or a translated background, is applied coherently across a clip, and the proxy measures how much an untrained network's internal features move in response. Our mechanistic guess comes from MoViNets, which used architecture search to find streaming video networks and discovered that early layers benefit from spatio-temporal kernels while deeper layers prefer spatial ones. Early mixing across frames can cancel slow illumination drift, much like frame differencing, but it can amplify camera jitter. If so, robustness should be readable from the network's response to coherent temporal perturbations. Independent per-frame noise, which is what 2D proxies effectively use, should miss it.
Relation to Prior Work
Three papers motivate the proposal because they show robustness can be scored cheaply on images. CRoZe perturbs both weights and pixels of an untrained network in a single step and measures how consistent its features, parameters and gradients stay. It reports a Spearman correlation of about 0.97 with trained robust accuracy on NAS-Bench-201 and DARTS spaces. The proxy $R$ in Zero-cost Proxy for Adversarial Robustness Evaluation combines kernel eigenvalues with a curvature estimate. It reports Kendall's tau of 0.33 on its 223-model Tiny-RobustBench and admits a bias against skip connections. Jung et al. supply the "stacking" finding: robust image cells stack convolutions rather than using wide parallel paths. All three work on static 2D cells and image corruptions. We keep their idea of scoring consistency under perturbation and change what is perturbed: whole clips along time, inside 3D and causal blocks.
That skip-connection bias, together with the stacking finding, also constrains the design. Efficient streaming networks depend on residual connections and causal frame buffers, so a proxy that penalizes skips may push search toward networks that cannot be deployed. The benchmark measures directly whether the stacking pattern holds for video.
On method, NAS-Bench-Suite-Zero shows how to compare 13 proxies fairly in one codebase. It finds that proxies carry complementary information and that FLOPs alone is surprisingly strong; we adopt its implementations and its correction for model size. Rethinking Performance Estimation in NAS shows that short, low-resolution training preserves rankings on clean images. We test whether that rule survives when the signal separating hand from clutter lives in fine edges.
On protocol, Goyal et al. show that fixing augmentation and model selection lifted PointNet++ from 90.0% to 93.3% on ModelNet40. They also show that reporting peak test accuracy across epochs leaks test information. Zhang et al.'s RGB-D survey documents inconsistent subject splits and saturated lab benchmarks. Together these explain why gesture numbers currently cannot be compared. Reported results on EgoGesture use different metrics and modalities: 92.48% RGB clip accuracy for MTUT, 94.04% depth clip accuracy and 91.04% streaming Levenshtein accuracy for Köpüklü et al., and incremental accuracy for CatNet. Our benchmark does not compete with those systems. It holds everything except architecture fixed.
Research Program
Search space. The space follows MoViNets' MobileNetV3-style macro design. For each stage, the choices are the temporal kernel size (1, 3 or 5 frames), causal or non-causal temporal padding, width, depth, and whether the block has a residual connection. We sample about 400 architectures stratified by FLOPs between roughly 0.3 and 3 GFLOPs per clip, which is the regime relevant to edge devices.
Frozen protocol. Every model is trained on RGB EgoGesture with the official subject split. Inputs are 16 frames at 112 pixels, with one fixed augmentation recipe, one optimizer schedule and one epoch budget. Checkpoints are selected on the validation split only, so the test set never influences model choice. Each architecture gets one seed. A random subset of 40 gets three seeds, so we can estimate seed noise and compare it with the spread across architectures. Our rough estimate is 1,500–2,500 GPU hours. We report this cost as part of the contribution, because the resulting lookup table replaces it for everyone afterwards.
Corruption suite. Each temporally coherent corruption is paired with a per-frame control of matched pixel-level magnitude. The coherent corruptions are:
- illumination ramps and flicker
- a cast shadow sweeping across the frame
- a moving skin-toned distractor patch
- camera shake as small random affine jitter, plus a smooth ego-motion pan
- motion blur
The controls are independent per-frame brightness changes and sensor noise. Each corruption comes at five severities, following the CIFAR-10-C convention, and is applied only at test time. Where EgoGesture's scene annotations allow, we add a real shift by holding out scenes, training on indoor scenes and testing on outdoor ones. A smaller replication runs on nvGesture, which was recorded under varying lighting.
Temporal consistency proxy. Let $f_\ell(x)$ be the layer-$\ell$ features of an untrained network on clip $x$. Let $\delta^{\text{coh}}$ be a perturbation sampled from the coherent corruption families, and $\delta^{\text{iid}}$ its per-frame counterpart at matched energy. The score is
$$ S(f) = \frac{1}{L}\sum_{\ell=1}^{L} \cos\!\big(f_\ell(x),\, f_\ell(x+\delta^{\text{coh}})\big) \;-\; \lambda \,\frac{1}{L}\sum_{\ell=1}^{L} \cos\!\big(f_\ell(x),\, f_\ell(x+\delta^{\text{iid}})\big), $$
where $L$ is the number of blocks and $\lambda$ is a single weight fit on a held-out 10% of the benchmark.
The subtracted term is the key ingredient. It removes generic noise insensitivity, which tracks network size and smoothing, and keeps only what is specific to time. To test whether the proxy simply rewards removing skips, every analysis is also reported within residual and non-residual subgroups.
Baselines
Naive references. The do-nothing references rank architectures by FLOPs, by parameter count, and by clean validation accuracy. NAS-Bench-Suite-Zero found FLOPs among the most robust single proxies, so the size-based rankings are serious contenders. Clean accuracy is the most important of the three. If clean-accuracy rank already predicts corrupted-accuracy rank with Kendall's tau above about 0.85, then robustness-specific search adds little and our first claim is weakened. All of these are free once the benchmark exists.
Standard proxies. The 13 proxies from NAS-Bench-Suite-Zero are computed with its NASLib implementations, adapted to 3D convolutions. We also test its XGBoost model that combines all proxies, fit on the same 10% calibration split as our $\lambda$.
Strongest existing method. CRoZe, applied unchanged frame by frame (and to the 3D network on clips with its usual single-step perturbation), is the strongest competitor. It is the only published proxy aimed at natural corruptions, and its reported image-domain correlation of about 0.97 is very high. Proxy $R$ is included for completeness; its reported tau of 0.33 is lower but was measured on adversarially trained models. No reported result was found on Lacuna for any proxy's correlation with corrupted video accuracy, or for corruption-robust accuracy on EgoGesture.
Matched-compute low-fidelity training. Following BPE, each candidate is trained briefly at reduced epochs and resolution, at two budgets matched to 10× and 100× the wall-clock cost of our proxy.
Ablation. Setting $\lambda=0$ isolates the per-frame-control subtraction. Using only per-frame perturbations isolates the role of temporal coherence.
Beating CRoZe means a higher Kendall's tau with mean corrupted accuracy on the 90% evaluation split, with non-overlapping bootstrap intervals. It also means higher recall of the true top-10% robust architectures, since proxies are known to struggle in the top percentiles. Both must hold within residual and non-residual subgroups.
Why not just…?
Why not just augment a fixed backbone with synthetic lighting and shake? This is the most serious threat. Goyal et al. showed that protocol alone can close apparent architectural gaps. We train a stratified 60-architecture subset a second time with the corruption families added to the augmentation, holding out one family for testing. If the spread across architectures in corrupted accuracy collapses below seed noise after augmentation, architecture search for temporal robustness is not worth doing. We would report that as the main finding and stop recommending it. Jung et al.'s 20–41% spread suggests the spread will survive, but that evidence is from images and adversarial attacks, so the experiment must decide it.
Why not just use CRoZe or another image proxy unchanged? CRoZe perturbs single images in one step. It cannot tell a network that cancels a slow brightness ramp from one that does not, because that difference only appears across frames. The per-frame-only ablation and the head-to-head with CRoZe test this directly. If CRoZe matches our proxy, the temporal ingredient is unnecessary and we say so.
Why not just train each candidate briefly instead? BPE-style short training may well rank better than any proxy at initialization. The question is cost: video training is far more expensive than CIFAR training. We compare at matched wall-clock budgets and separately test BPE's rule that resolution can be downscaled. If downscaling to 56 pixels destroys rank preservation under the shadow and distractor corruptions but not on clean clips, that is a concrete, reusable warning for video search.
Why not just collapse clips into motion-history images and use a 2D network? The benchmark includes a motion-history-image plus 2D-CNN family at matched FLOPs, as a point outside the search space. If it is competitive under coherent interference, the case for searching 3D streaming networks weakens accordingly.
Proposed Experiments and Decisive Checks
The first check is variance decomposition. Across the 40 multi-seed architectures, we compare the seed variance of corrupted accuracy with the variance across architectures, both before and after regressing out clean accuracy and FLOPs. If architecture explains less than seed noise once those are removed, the first claim fails.
The second check is proxy ranking on the held-out 90%. We report Kendall's tau and top-10% recall for our proxy, CRoZe, the other 13 proxies, the combined XGBoost model, and the FLOPs and clean-accuracy references. Results are given per corruption family, and we expect our proxy's advantage to concentrate on coherent corruptions rather than per-frame ones. If the advantage is uniform across families, our proxy is just a better noise-sensitivity score, not a temporal one.
The third check tests the stacking question in video. We regress corrupted accuracy on residual connections and early temporal kernel size, controlling for FLOPs. This shows whether Jung et al.'s pattern holds when skip connections are needed for streaming efficiency.
The fourth check is transfer. A proxy-guided search under a 1 GPU-day budget picks architectures, and we train the top five on EgoGesture and on nvGesture. They are compared with FLOPs-matched architectures picked by clean accuracy, using streaming Levenshtein accuracy under the frozen protocol. This makes the comparison against Köpüklü et al.'s metric meaningful rather than anecdotal.
The benchmark, the training recipe and the corruption code will be released together. The intent is that later gesture papers report numbers that can actually be compared.
References
- Neural Architecture Design and Robustness: A Dataset
- MoViNets: Mobile Video Networks for Efficient Video Recognition
- Generalizable Lightweight Proxy for Robust NAS against Diverse Perturbations
- Zero-cost Proxy for Adversarial Robustness Evaluation
- NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies
- Rethinking Performance Estimation in Neural Architecture Search
- Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline
- RGB-D-based action recognition datasets: A survey
- Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition With Multimodal Training.
- Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks
- CatNet: Class Incremental 3D ConvNets for Lifelong Egocentric Gesture Recognition