In a video, most of each frame is a near copy of the last one. A vision model that runs every image patch through every layer on every frame therefore redoes most of its work. Mixture-of-experts models are networks whose large feed-forward layers are split into many smaller "experts," with a small learned scorer, the router, sending each patch to one or two of them. They already separate total model size from per-patch compute. They still treat each frame as if it had never been seen, though. Their router looks at a patch in isolation and cannot tell whether that patch's content was obvious from the previous frame. Work on emergent attention points to a different signal. When a model continually predicts its next input, the predictable parts (static background, smooth camera motion) cancel out, and the leftover prediction error lands on the things worth looking at, such as moving actors.
The proposal is to put these together for streaming video. A cheap per-patch predictor guesses each patch's features at a mixture-of-experts layer from the previous frame. Only patches whose prediction error is large relative to the predictor's own stated uncertainty get fresh expert computation. Every other patch reuses its cached expert output from the previous frame. Scaling the error by uncertainty is what keeps this from failing in the obvious way. Raw surprise is highest on pure noise, such as a flickering screen, rain, or leaves, so a router driven by raw error would spend its whole budget on meaningless pixels. The claim is that uncertainty-scaled error is a better and more stable compute-allocation signal than a learned router, frame differencing, or batch-level priority. As a side effect, the map of which patches received compute should be a usable unsupervised localization of the action.
Core Claim
For a streaming video model with sparse expert layers, we make three claims. First, the decision of whether a patch needs new expert computation should come from how much its current features deviate from a temporal prediction, scaled by the predicted uncertainty. Second, skipped patches should reuse cached outputs rather than be discarded. Third, at a fixed compute budget, this beats routers that see only the current frame, and it beats simple change detection whenever the camera moves or the scene contains unstructured motion.
Concretely, let $z_{i,t}\in\mathbb{R}^d$ be the input to an expert layer for patch $i$ at frame $t$. A small predictor reads frame $t-1$ at that layer and outputs a mean $\hat z_{i,t}$ and a per-dimension variance $\sigma^2_{i,t}$. The priority score is
$$s_{i,t}=\frac{1}{d}\sum_{j=1}^{d}\frac{(z_{i,t,j}-\hat z_{i,t,j})^2}{\sigma^2_{i,t,j}} .$$
The predictor is trained by Gaussian negative log-likelihood, so a well-calibrated predictor gives $s\approx 1$ on regions it has correctly judged to be noisy. The score rises well above 1 only where the world did something the predictor considered unlikely. This is a working stand-in for the "Bayesian surprise" that Salience and Attention in Surprisal-Based Accounts of Language Processing argues biological attention tracks, as opposed to raw information content. That review resolves the snow-screen paradox, the observation that static noise is maximally surprising pixel by pixel yet humans ignore it, only conceptually. Here it becomes a measurable design choice.
Within each frame, the top fraction $\rho$ of patches by $s_{i,t}$ is sent through the standard router, which chooses which expert to use. The rest skip the expert layer and reuse their output from frame $t-1$. Because the budget is set per frame, there is no batch dependence and the model stays causal.
Relation to Prior Work
The closest work is Surprisal-Triggered Conditional Computation with Neural Networks. It uses the surprisal of a frozen autoregressive speech model to switch between a small and a large network. It matched an always-on large model on TIMIT with 15% fewer FLOPs, and it had lower error variance than a learned controller. That paper motivates our choice of signal: its result is the main evidence that surprisal is a more stable gate than a learned controller. We differ in four ways. Our decision is per patch, not per input. Skipped work is replaced by cached output instead of a weaker network. The surprisal is scaled by uncertainty instead of raw. And the predictor is trained jointly with the backbone instead of kept frozen, which that paper leaves open.
Two mixture-of-experts papers supply the method we build on, and each shows a limitation. Scaling Vision with Sparse Mixture of Experts introduced batch-prioritized routing. It ranks image patches by router score and drops low-priority ones, and it matched dense ViT-H with under half the FLOPs on ImageNet. Its priority comes from a router trained on classification and has no temporal context. Mixture-of-Experts with Expert Choice Routing gives variable compute per token by letting experts pick tokens. That choice needs visibility across a whole batch, which the research file flags as awkward for streaming, token-by-token inference. We keep their routers for choosing which expert and replace only the decision of whether a patch is computed at all.
The emergent-attention papers supply the empirical premise. Action Localization through Continual Predictive Learning showed that errors from predicting future frame features concentrate on human actions and ignore camera shake. Actor-Centered Representations for Action Localization in Streaming Videos pushed this to single-pass streaming and reached 0.70 mAP on UCF Sports without box labels. Both use error to localize. Neither uses it to allocate compute inside a large model.
The same papers also set constraints. The second reports attention drift in multi-actor scenes when an irrelevant entity moves unpredictably. Deep Predictive Coding Network with Local Recurrent Processing for Object Recognition shows that iterative error-correction loops add sequential latency. We therefore deliberately use a single feed-forward prediction step and no recurrent settling.
Research Program
We will build a video transformer in which every other feed-forward layer is a 16-expert, top-1 or top-2 layer. Each expert layer gets a small predictor: a depthwise temporal convolution plus a two-layer network over the previous frame's patches and their spatial neighbors, costing under 5% of an expert's FLOPs. The predictor is trained with the Gaussian likelihood above, and its gradient is stopped into the backbone so that the predictor cannot make features trivially predictable. The rest of the network is trained on the task loss. Training uses the hard top-$\rho$ selection with a straight-through estimate, and $\rho$ is sampled per clip so that one model covers a whole compute curve.
The analysis has three threads. The first is the accuracy-versus-compute curve as $\rho$ goes from 1 down to 0.1. The second is whether high-$s$ patches line up with annotated actors, which tests whether "emergent attention" falls out of the compute allocation. The third is expert behavior: utilization entropy, and whether removing the auxiliary load-balancing loss becomes safer once the always-predictable patches are filtered out. That last question tests the collapse-versus-homogenization tension raised in Advancing Expert Specialization for Better MoE. We treat it as secondary, because we do not expect error filtering on its own to cure collapse.
Baselines
The do-nothing reference is the same mixture-of-experts video transformer computing every patch on every frame ($\rho=1$). It sets the accuracy ceiling and the full-compute cost. Alongside it we report uniform random skipping with cache reuse at each $\rho$. This separates the value of the selection signal from the plain fact that cached outputs from adjacent frames are often good enough. If random skipping comes within noise of our method, the error signal adds nothing.
The strongest existing method for patch-level priority is batch-prioritized routing from Scaling Vision with Sparse Mixture of Experts. We apply it within each frame, ranking by router score, and give it the same cache reuse so that the only difference is the ranking signal. The file reports no streaming-video number for it. Its reported figure is 90.35% ImageNet top-1 for V-MoE-15B, with dense-ViT-H parity at under 50% of the FLOPs, which is the size of efficiency gain we must at least match in video. We also run expert-choice routing (Mixture-of-Experts with Expert Choice Routing), restricted to one frame as the batch, as the strongest variable-compute router. The surprisal-triggered method (Surprisal-Triggered Conditional Computation with Neural Networks) is reimplemented at frame level, deciding whether a frame gets full computation or cache reuse, to test whether patch-level granularity matters. The file reports no video result for it; its reported number is 15% fewer FLOPs at matched quality on TIMIT.
On task quality, the strongest number in the file is 0.70 mAP at 0.2 overlap on UCF Sports for unsupervised streaming localization, from Actor-Centered Representations for Action Localization in Streaming Videos. We compare to it only in the unsupervised track described below.
Every compute-allocating baseline shares the backbone, the data, the training schedule, the expert count, and a matched hyperparameter search of 20 trials per method. All are compared at identical measured per-frame FLOPs and wall-clock time on the same hardware, and the cost of the predictor is charged to our method. Beating a baseline means a higher accuracy-versus-compute curve over at least the range $\rho\in[0.2,0.5]$, with non-overlapping three-seed intervals.
Two ablations isolate the key ingredients. Raw error (setting $\sigma\equiv1$) versus uncertainty-scaled error isolates the noise-robustness claim. A frozen predictor versus a co-trained one isolates the co-adaptation question left open by the surprisal-triggered work.
Why not just…?
Why not just skip patches whose pixels or features barely changed since the last frame? Frame differencing costs almost nothing and is the most serious threat on footage from a static camera. We expect it to match our method there, and if it does across the board, the predictor is not worth keeping. It should break when the camera pans or shakes, because then every patch changes. Prior predictive-learning work found that this kind of change is exactly what temporal prediction cancels (Action Localization through Continual Predictive Learning). We include a feature-difference baseline and report results separately on clips with static and moving cameras. That split settles the question directly.
Why not just give the ordinary learned router the previous frame as extra input? This is a real threat. A router that sees $(z_{i,t}, z_{i,t-1})$ could in principle learn the same computation we hand-specify. The case for an explicit predictor rests on the surprisal-triggered result, where a learned gate was less stable and more sensitive to hyperparameters than a surprisal gate. It also rests on the fact that a task-trained router has no built-in notion of calibrated noise. We run this temporal-context router with a matched parameter count and tuning budget. If it equals our method, including on the injected-noise test below, we will report that the benefit comes from temporal context rather than from prediction error.
Why not just use batch-prioritized or expert-choice routing, which already vary compute per patch? Both rank patches by a single-frame, task-trained score. Neither can know that a patch is redundant given the past. Expert choice additionally needs a batch to rank over, which the research file identifies as a problem for streaming inference. Both are run as baselines under identical cache reuse, so any gain we show comes from the ranking signal alone.
Proposed Experiments
The first track is supervised streaming recognition and localization on UCF Sports and JHMDB, plus multi-actor group activity on the Collective Activity Dataset. For the latter, Actor-Centered Representations for Action Localization in Streaming Videos reports 80.2% accuracy after one fine-tuning epoch, which serves as a reference point. We plot accuracy and mAP against per-frame FLOPs and wall-clock time. Zero-shot transfer from sports videos to THUMOS'13, where the same paper reports 0.38 mAP, tests whether the allocation policy generalizes beyond its training domain.
The second track is unsupervised localization. Without box labels, the $s$ map is combined with class-agnostic proposals following the protocol of Action Localization through Continual Predictive Learning and scored against the reported 0.70 mAP on UCF Sports. Matching it while also serving as a compute allocator would show that one signal can do both jobs. Falling well short would mean the scaled error is a worse localizer than raw error, which would be informative in its own right.
The decisive check is a noise-injection test. We paste dynamic, unlearnable noise into 10–30% of the area of UCF Sports and JHMDB clips: animated static, flickering textures, and synthetic rain. We then measure the fraction of the compute budget each method spends inside the noise and the resulting drop in accuracy. The prediction is specific. Raw-error routing and frame differencing should send a large share of their budget into the noise and lose accuracy at low $\rho$. Uncertainty-scaled routing should learn high $\sigma$ there and keep its budget on actors. If uncertainty-scaled routing spends roughly as much budget on noise as raw-error routing does, the central mechanism has failed. A second check targets drift in multi-actor scenes: on Collective Activity, we measure how much compute goes to non-participating movers during the first second of each clip, compared with the temporal-context router. Finally, we report the latency of the predictor itself, so that the efficiency claim holds in wall-clock time and not only in FLOPs.
References
- Salience and Attention in Surprisal-Based Accounts of Language Processing
- Surprisal-Triggered Conditional Computation with Neural Networks
- Scaling Vision with Sparse Mixture of Experts
- Mixture-of-Experts with Expert Choice Routing
- Action Localization through Continual Predictive Learning
- Actor-Centered Representations for Action Localization in Streaming Videos
- Deep Predictive Coding Network with Local Recurrent Processing for Object Recognition
- Advancing Expert Specialization for Better MoE