Length-Aware Hardware Budgets for Long Video Generation

Long videos are usually generated in chunks: the model produces a second or two of footage, stores what it has seen in a memory of past attention keys and values (the "KV cache"), and conditions the next chunk on that memory. Because the memory grows with every chunk, so does the cost of producing each new chunk. The bottleneck also moves as the video gets longer. Early in a rollout, generation is limited by arithmetic. Later it is limited by memory capacity and bandwidth, and on a 24 GB consumer GPU it eventually runs out of memory altogether. Yet every efficiency recipe for long video fixes one configuration and uses it for the entire video: one window size per attention head, one sparsity ratio, one cache precision. Hardware-aware architecture search is the field that picks network configurations using measured device cost instead of FLOP counts. So far it has optimized a single forward pass at a fixed input size.

We propose to search for a horizon-indexed schedule. For each attention head and each phase of the rollout, the schedule says how much past context the head sees (local tile window, number of retrieved past chunks) and at what cache precision. The search must satisfy a hard per-chunk latency deadline and a hard memory cap on a specific device. The claim is that, at an equal per-chunk budget, redistributing context across heads as the video lengthens keeps subject identity and motion coherent for longer than any fixed configuration. It also claims that the right schedule differs measurably between a compute-rich H100 and a memory-poor RTX 4090.

Core Claim and Why It Is Not Already Done

The nearest work all searches or tunes attention structure, but at a single sequence length and a single operating point. Fast Video Generation with Sliding Tile Attention searches per-head window spans under a fixed tile shape and builds a GPU kernel that turns sparsity into wall-clock speed. Veda picks a 3D tile shape per head by minimizing reconstruction error against full attention, at a fixed global sparsity. FPSAttention shows that different denoising steps tolerate different amounts of compression, but it sets this schedule by hand and studies fixed-length clips. S2DiT is the only work that runs an explicit budget-constrained search for streaming video. It searches the ordering of two fixed block types via dynamic programming against mobile latency, and it falls back to window attention when its cache grows.

These works motivate the proposal: head specialization is real, and step-dependent tolerance is real. They also supply its parts: tile-aligned kernels, retrieval over past chunks from Mixture of Contexts, and low-bit cache quantization from Quant VideoGen. What none of them does is treat the rollout position as a variable of the architecture, or enforce a device budget that must hold at frame 1,400 rather than frame 32.

The constraint side borrows from ConNAS. ConNAS treats hardware limits as hard boundaries and steers candidates back toward feasibility, which gave 100% constraint satisfaction where penalty-based searches failed. Our budget is a deadline per chunk, for example finishing each chunk within its own playback duration for real-time streaming. A soft average-cost penalty would let late chunks miss that deadline, so hard constraints are necessary here.

Research Program

The first component is a length-aware cost model. HW-NAS-Bench documents that additive per-layer lookup tables ignore fusion, caching and launch overhead. In our setting they also miss the dependence on context length: dense attention cost grows with the square of history, a tile window stays constant, and a quantized cache trades bandwidth for dequantization work. We will measure attention-kernel latency and memory for each head option across a grid of context lengths on each device. We will then fit a model of the form

$$c_d(o, L) = a_{d,o} + b_{d,o}\,L + e_{d,o}\,L^2,$$

where $c_d$ is the measured cost on device $d$ of head option $o$ at context length $L$ tokens. The coefficients $a$, $b$, $e$ are fitted per device and option, and the quadratic term is there because dense attention over history is quadratic in $L$. A small learned correction will capture interactions within a layer. Before the cost model is used for search, it has to beat the additive table on rank correlation with measured end-to-end chunk latency.

The second component is a rollout-aware quality signal. Following Veda's teacher idea, we run the frozen model with full attention over real long video latents. At several rollout positions we record how much each head's output changes under each option. This gives an error $\epsilon_h(o, p)$ for head $h$, option $o$ and rollout phase $p$. The open empirical question is whether $\epsilon$ actually varies with $p$. One possibility is that some heads only start needing distant context once identities have been established and leave the frame.

The third component is the search. Within each phase $p$, the search chooses one option per head to minimize total error subject to the latency and memory caps. Because per-head costs add within a fused layer once the learned correction is included, this is a multiple-choice knapsack problem. It is solved exactly with a Lagrangian relaxation plus a ConNAS-style feasibility repair whenever measured cost exceeds the deadline. Phases are coupled by the cache: a precision chosen for stored keys and values persists. The search therefore runs forward over phases while tracking cache memory, and back-tracks if a later phase becomes infeasible.

The fourth component is a short fine-tune of the base model under the found schedule. STA shows that fine-tuning moves its speedup from 1.89x to 3.53x. We use a small fixed budget, identical for every searched or hand-set configuration.

The base model is the chunk-autoregressive 8B model HY-WorldPlay-8B, which Quant VideoGen ran on a single RTX 4090. The two devices are an H100 and an RTX 4090. They are chosen because their binding constraints differ, which tests whether the schedule is genuinely device-specific.

Baselines

The naive reference is the unmodified model with full-precision dense attention over all history. It runs until it exhausts memory, and the frame count at which it does so is reported. On the RTX 4090 it bounds what "doing nothing" achieves. It also serves as the quality ceiling for the frames it can produce.

The strongest existing method for long rollouts under a memory cap is Quant VideoGen: dense context with 2-bit cache quantization. It reports up to 7.0x cache compression, generation to 1,400 frames, under 4% latency overhead, and PSNR around 28.7 against roughly 20–21 for standard LLM cache quantizers (Quant VideoGen). Its per-chunk latency still grows with history, so it is the natural competitor for "longest coherent video on a 4090." Beating it means two things together: holding a fixed per-chunk deadline it cannot meet past some horizon, and matching its PSNR and consistency at every horizon where both run.

The strongest fixed sparse configuration is STA's per-head window search, run once at the short training length and then held fixed (Sliding Tile Attention). We combine it with Quant VideoGen's cache quantization so that it, too, fits the memory cap. Its reported numbers are 1.89x training-free and 3.53x fine-tuned end-to-end speedup, measured on HunyuanVideo and Wan 2.1, not on our model. Lacuna has no reported result for STA on an autoregressive long-rollout model. We will rerun it ourselves with the same kernels, the same fine-tune budget and the same deadline. We also include Mixture of Contexts with a fixed top-k retrieval, tuned to meet the same deadline. MoC reports a 2.2x end-to-end speedup on minute-scale sequences (Mixture of Contexts).

Two ablations isolate the key ingredient. The first is a static schedule searched with the same machinery at the longest horizon: one configuration, chosen to be feasible at the final chunk and used everywhere. The second is the same search driven by an additive lookup-table cost model instead of the length-aware one, counting how often the result breaks the deadline when actually measured. If the static ablation matches the horizon-indexed schedule, the core claim fails.

Why not just…?

Why not just cap context at a fixed sliding window so cost never grows? This is the most serious threat, because a constant window trivially meets any deadline. The weakness is that it permanently discards distant history. MoC and the drift literature report that identity loss across chunks is exactly what this causes. Our experiment settles it directly: at an identical measured per-chunk latency, we compare subject consistency in late time bins (frames 1,000–1,400). If the fixed window ties, we report that the search adds nothing.

Why not just run STA's or Veda's per-head search once and keep it? Both are strong and both are fixed-length. The static-at-longest-horizon ablation is exactly this idea, upgraded to be deadline-feasible. The proposal earns its keep only if head error $\epsilon_h(o,p)$ shifts with $p$ enough that a single assignment is either infeasible early-wasteful or quality-poor late. We will measure that shift before running any search. If it is negligible, we drop the horizon index and report a deadline-feasible static search as the contribution.

Why not just apply uniform low-bit cache quantization everywhere? Quant VideoGen shows this works well for memory. However, latency still grows with history, and FPSAttention shows that precision and sparsity interfere on high-magnitude attention entries. Uniform precision therefore cannot be combined freely with sparsity. Letting precision vary per head and phase is how the search can route around that interference. The Quant VideoGen baseline tests whether it actually helps.

Why not just use differentiable search with a latency penalty? ConNAS and DUCCIO document that penalty weights need trial-and-error tuning and that the results frequently violate budgets. A per-chunk real-time deadline is a hard constraint, so we report violation rates for a penalty variant given the same search budget.

Proposed Experiments

Evaluation uses prompts from the MovieGen prompt suite and VBench (VBench), with rollouts to 1,400 frames. VBench subject consistency, motion smoothness and temporal flickering are reported per time bin rather than averaged, so that drift is visible. PSNR is measured against the dense reference rollout wherever that reference fits in memory, following Quant VideoGen's protocol. Hardware metrics are measured, never estimated: P99 per-chunk latency against the deadline, peak memory, and the horizon at which each method first misses the deadline or runs out of memory.

The first decisive check is the cost model. Its Kendall correlation with measured chunk latency across context lengths must clearly exceed that of the additive table, and the additive-table search must produce measured deadline violations that ours avoids.

The second decisive check is the late-horizon quality comparison at matched latency. The horizon-indexed schedule must beat the fixed window, the fixed STA-plus-quantization configuration, and the static-search ablation on late-bin subject consistency, with no loss in early bins.

The third check is device specificity. We swap the schedules searched for the H100 and the 4090 and measure the loss on each device. If the swap is free, the hardware-aware part of the claim is unsupported and we will say so. Human side-by-side preference against Quant VideoGen on late segments completes the evaluation.