Video analytics systems save compute by adapting how they process each stretch of video. They lower the resolution on easy segments, sample fewer frames when little is moving, and spend full effort only where the content demands it. These systems almost always assume that a cheaper configuration makes the whole job proportionally faster. On a GPU that assumption is wrong in a specific way. The time to process a segment depends on its own configuration and also on how many other segments share that configuration, because a GPU is only efficient when it runs large batches of identically shaped tensors. A policy that gives twenty segments twenty slightly different "optimal" settings can run slower than one that gives all twenty the same moderately expensive setting.
This proposal treats configuration choice for a window of video segments as a single joint decision. A cheap, fully parallel probe looks at the whole window. An optimizer then assigns each segment one of a small set of precompiled tensor shapes. It maximizes predicted accuracy under a wall-clock budget, and the cost model is measured latency as a function of how many segments land in each shape. In short, content adaptation should decide which segments share work, not only how little work each segment gets.
Core Claim
In content-adaptive video inference, per-segment cost is not additive. A configuration's marginal cost is high when few segments use it, because the batch is small, the GPU is underused, and there is fixed launch overhead. The cost is low when many do. Our claim is that the gap between the speedup a content-adaptive system predicts from FLOPs and the speedup it actually delivers comes largely from this non-additivity and from sequential decision-making. We claim further that an assignment procedure that optimizes accuracy against a batch-dependent latency table recovers most of that gap. It should beat both FLOP-proportional adaptive policies and uniform, fully batched processing on measured milliseconds at matched accuracy.
Where This Sits
The content-adaptation side supplies the application regime and the knobs. Zeus uses a Deep Q-Network, a reinforcement learning agent that scores actions by expected future reward, to pick resolution, segment length, and sampling rate per segment for action localization. It reports a 3.4x average speedup over sliding windows. Its authors state that the agent's sequential decisions rule out intra-video parallelism. Skyscraper plans and switches knobs for streaming ingestion using coarse server-level cost. AdaFrame picks frames one at a time with a recurrent network and counts savings in frames and GFLOPs. All three motivate this work, since each shows that content adaptation pays off. All three also share the blind spot we target: cost is modeled per segment and independently of how execution is batched. We reuse Zeus's knob space and query setup and replace its decision procedure.
The FLOPs-latency side supplies the constraint and part of the method. ShuffleNet V2 shows that fragmentation and memory traffic, not arithmetic, set speed. NetAdapt replaces proxies with measured per-layer lookup tables on the target device. We reuse that measurement philosophy but index the table by configuration and batch size instead of by layer. Latency-aware Spatial-wise Dynamic Networks and LAUDNet are the closest work. They show that dynamic skipping only speeds up GPUs when decisions are coarse enough to keep memory contiguous, and they find a U-shaped relation between skipping granularity and latency. Both operate inside a single image. We make the analogous argument one level up, across segments of a video, where the "granularity" is how many distinct shapes a batch is split into. The two approaches are complementary: patch skipping could run inside each of our shape buckets.
Empowering Adaptive Early-Exit Inference with Latency Awareness optimizes exit thresholds under a latency budget but treats each layer's latency as a fixed scalar. The core of our proposal is relaxing exactly that assumption. Adaptive Token Sampling acts as a warning. Its variable token counts per input break standard batching and force padding, which is the failure our fixed shape set is designed to avoid. ACRoBat batches dynamic control flow at compile time for language models. We borrow its insight that dynamic programs must be regrouped into homogeneous batches, but we do the regrouping at the policy level, where accuracy trade-offs are visible, rather than in the compiler.
Research Program
The system has three parts.
Shape set. The first is a fixed set $C$ of roughly 8 to 16 configurations drawn from Zeus's knobs of resolution, segment length, and sampling rate. Each configuration is precompiled as its own engine with static shapes, which removes the dynamic-shape reallocation stalls the file documents. For each configuration $c$ and each batch size $n$ on each target GPU, we measure the wall-clock time $T_c(n)$. The cost is a few minutes of profiling per device, which directly addresses the transferability complaint against lookup tables.
Parallel probe. The second part replaces Zeus's sequential agent. Over a lookahead window of $W$ segments, a lightweight model processes all segments at the lowest configuration in one batch. A small head then predicts $a_{ic}$, the probability that segment $i$ is handled correctly under configuration $c$. The head is trained offline by running every configuration on training videos.
Assignment. The third part solves
$$\max_{x}\ \sum_{i,c} a_{ic}\, x_{ic} \quad \text{s.t.}\quad \sum_{c\in C} T_c\Big(\sum_i x_{ic}\Big) + T_{\text{probe}}(W) \le B,$$
where $x_{ic}\in\{0,1\}$ assigns segment $i$ to exactly one configuration $c$ and $B$ is the wall-clock budget for the window. Because $T_c$ is concave-like in batch size, opening a configuration carries an effective fixed cost. The problem therefore resembles facility location. With $|C|$ small, we solve it by Lagrangian relaxation over the budget combined with a local search over which configurations to open. We expect this to run in well under a millisecond per window, and we will measure that overhead and charge it to the method.
The analysis component asks when non-additivity matters. We will characterize the shape of $T_c(n)$ on server GPUs (V100, RTX 3090) and edge GPUs (Jetson TX2, Orin Nano). We predict that the method's advantage over FLOP-proportional policies grows with how early batch efficiency saturates on a device, so it should be largest on datacenter GPUs and smaller on edge devices. We also predict the advantage grows with content diversity in the window, because diverse content tempts a naive policy into many small batches. Both predictions can fail, and we report them either way.
Baselines
The naive reference point is uniform processing. Every segment runs at a single fixed configuration, fully batched, as in Zeus's sliding-window baseline. We sweep over all configurations in $C$ to trace an accuracy-versus-latency curve, and every adaptive method must lie above that curve to count as useful.
The strongest existing adaptive method is Zeus. Zeus reports 3.4x average and up to 4.7x faster query processing than sliding windows at the same target accuracy, and 8.3x faster with 0.25 higher F1 than frame-based filtering on its benchmarks (Zeus). The file gives no per-query latency numbers for THUMOS14 or ActivityNet queries, so the absolute comparison will come from our reimplementation. We will train it on the same queries and videos with the same backbone, the same shape set, and the same tuning budget. We will measure it in milliseconds on identical hardware. Beating Zeus means lower wall-clock latency at the same F1 target on a majority of queries across at least two GPUs, with the probe and optimizer overhead included.
Two ablations isolate the key ingredient. The first keeps our probe and optimizer but uses an additive cost $\sum_i t_c$ with per-segment latency measured at batch size one or scaled from FLOPs. Any gain of the full method over this ablation is attributable to batch-dependent cost. The second runs Zeus's decisions, or the additive ablation's, and then applies post-hoc bucketing: segments are grouped by chosen configuration and batched after the fact. This separates the gain from batching itself from the gain from making the policy aware of batching.
We also include a static, hardware-optimized backbone tuned NetAdapt-style to the budget and applied uniformly. An oracle that assigns configurations using true per-segment correctness serves as an upper bound.
Why not just…?
Why not just subsample uniformly and batch everything? Uniform processing avoids every irregularity our method has to manage, and on narrow content it may be hard to beat. The file's evidence cuts against it in general. AdaFrame matches full-frame accuracy on FCVID with 8.21 frames on average, while uniform sampling at similar budgets loses accuracy. Zeus and Skyscraper, the latter reporting up to 8.7x lower cost than a static baseline, report large gains over fixed configurations. Those gains are measured in their own cost units, though, so the uniform curve on real milliseconds is exactly what Experiment 1 re-establishes.
Why not just take an existing adaptive policy and bucket its decisions afterward? This is the most serious threat. Post-hoc bucketing recovers batching whenever the policy happens to concentrate on a few configurations. Our argument is that a policy blind to batch cost has no reason to concentrate, and that sending a borderline segment to an already-open configuration is often nearly free. If the post-hoc bucketing ablation comes within noise of the full method on both server and edge GPUs, the core claim is wrong. The contribution would then reduce to a measurement study of the FLOPs-latency gap in video adaptation, and we would report it that way.
Why not just use a static hardware-optimized network? NetAdapt and ShuffleNet V2 close the FLOPs-latency gap for a fixed workload, but they spend worst-case compute on every segment. The static-backbone baseline tests whether content adaptation adds anything once the backbone itself is fast. Our method can also sit on top of such a backbone, and we will run that combination.
Why not just do LASNet-style patch skipping inside each frame? Spatial skipping (Latency-aware Spatial-wise Dynamic Networks) attacks within-frame redundancy. Our method attacks across-segment configuration choice. They are orthogonal rather than competing, so we treat patch skipping as a possible extension and not as a baseline to beat.
Proposed Experiments
Experiment 1: size of the gap. We use the action-localization queries Zeus evaluates on BDD100K (CROSSRIGHT, LEFTTURN), THUMOS14 (POLEVAULT, CLEANANDJERK), and ActivityNet (IRONINGCLOTHES, TENNISSERVE). For each adaptive baseline, we measure its FLOP-predicted speedup and its wall-clock speedup over the uniform curve on V100, RTX 3090, and Jetson Orin Nano. Alongside latency, we record GPU utilization and the number of distinct configurations per window. If FLOP-predicted and measured speedups agree to within about 10% everywhere, the problem is small and the project stops there.
Experiment 2: the main comparison. Here we produce F1-versus-milliseconds curves for all methods in the Baselines section, including the two ablations, across budgets and window sizes $W \in \{8, 32, 128\}$. We also add ActivityNet recognition with AdaFrame's frame-selection setup, where AdaFrame reports 71.5% mAP with 8.65 frames. That setup tests whether the effect extends beyond localization queries.
Experiment 3: the decisive check. This experiment tests the mechanism rather than the headline number. On each device, we plot the full method's advantage over the post-hoc bucketing ablation against two quantities. The first is the batch size at which $T_c(n)/n$ saturates. The second is the content diversity of the window, measured as the entropy of the oracle's assignments. The claim predicts a positive relation with both. A flat relation, together with parity against post-hoc bucketing, would falsify it.
Experiment 4: robustness. We re-profile $T_c(n)$ under a different driver and batch-size regime and measure how much stale tables hurt. We also report probe and optimizer overhead as a share of total latency. Hyperion reports scheduling overhead under 3.5% of total latency (Hyperion), and we treat that figure as a reference point for acceptable overhead.
References
- Zeus: Efficiently Localizing Actions in Videos using Reinforcement Learning
- Extract-Transform-Load for Video Streams
- AdaFrame: Adaptive Frame Selection for Fast Video Recognition
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
- NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
- Latency-aware Spatial-wise Dynamic Networks
- Latency-Aware Unified Dynamic Networks for Efficient Image Recognition
- Empowering Adaptive Early-Exit Inference with Latency Awareness
- Adaptive Token Sampling for Efficient Vision Transformers
- ACROBAT: Optimizing Auto-batching of Dynamic Deep Learning at Compile Time
- Hyperion: Low-Latency Ultra-HD Video Analytics via Collaborative Vision Transformer Inference