Deadline-Aware Quantization of Learned Video Encoder Shortcuts

Modern video encoders such as Versatile Video Coding (VVC) get their compression gains by searching an enormous tree of ways to cut each frame into blocks. To make that search affordable, researchers train small neural networks that predict which block splits are worth trying and tell the encoder to skip the rest. These "partition predictors" have only ever been run in full floating point, inside reference software, on desktop CPUs. Deploying them in a real-time hardware encoder means shrinking them to a few bits per weight and running them on an accelerator with tight memory and timing budgets. Nobody knows what that does to the encoder.

This note proposes measuring, and then optimizing, quantization loss for these predictors in the units that matter to the encoder: bits wasted and per-block deadlines missed, as measured on the physical board. Classification accuracy is the wrong unit. The central observation is that quantization noise in a partition predictor matters only when it flips a skip decision. The two kinds of flip are not equally bad. Wrongly skipping a split permanently costs bitrate. Wrongly keeping a branch costs only time, unless that time blows the deadline. So the right number of bits cannot be chosen apart from the decision threshold. It also cannot be chosen from a simulator's latency estimate if that estimate ranks configurations differently from the real device.

Core Claim

For a learned encoder-control network, we can find a per-layer bit-width assignment $b$ and an early-skip threshold $Th$ that together solve

$$\min_{b,\,Th}\ \Delta\mathrm{BD\text{-}rate}(b, Th) \quad \text{s.t.} \quad \Pr\big[t_{\mathrm{CTU}}(b, Th) > \tau\big] \le \epsilon .$$

Here $\Delta\mathrm{BD\text{-}rate}$ is the Bjøntegaard-delta bitrate increase over the unaccelerated encoder, the standard measure of extra bits needed for equal quality. $t_{\mathrm{CTU}}$ is the measured time to decide one 128×128 coding tree unit (CTU, the encoder's top-level block): predictor inference on the accelerator plus the rate-distortion search the predictor leaves for the encoder to do. $\tau$ is the per-CTU budget for the target frame rate, and $\epsilon$ is the tolerated deadline-miss rate.

For 4K at 30 fps, a frame holds about 506 CTUs. That arithmetic gives $\tau$ of roughly 66 µs per CTU if CTUs are processed serially. This figure is a derived target, not a reported result.

The claim has three falsifiable parts. First, joint selection of $(b, Th)$ beats quantizing first and tuning the threshold afterward. Second, at the same deadline it beats mixed-precision search driven by a simulator. Third, quantization-induced decision flips concentrate on the skip-split side, near the threshold, and at small block sizes. If that third part holds, flips are the mechanism, and a loss that penalizes them asymmetrically can largely prevent them.

Grounding in Prior Work

Two predictors serve as test subjects, and both supply the method we reuse.

DeepQTMT is a six-stage early-exit network for intra-coded frames in VVC. It is trained with a loss that weights errors by their rate-distortion cost, and it reports 44.65% to 66.88% encoding-time savings at a 1.32% to 3.19% BD-rate cost. MBMP-CNN is a three-stream network for inter-coded frames. It predicts a split-depth map per CTU and uses a tunable threshold $Th$ to skip searches. It reports 17% to 30% time savings at a 0.37% to 1.18% BD-rate cost under Random Access, with inference taking 0.21% of encoder runtime. Both run in floating point inside the reference software and consider no hardware. The research file notes that the effect of quantization noise on their predictions is entirely unstudied. We keep their architectures and DeepQTMT's rate-distortion-aware loss. We change the numeric format, the deployment target and the objective.

HAQ is the closest existing work. It uses reinforcement learning to allocate per-layer bit-widths under latency and energy budgets taken from hardware simulators (BISMO for FPGAs, BitFusion for ASICs). It reports 1.4× to 1.95× latency reduction over fixed 8-bit on ImageNet classifiers. We differ from HAQ in three ways. Our reward is downstream encoder bitrate under a deadline, not top-1 accuracy. Our cost comes from board measurement, with simulator cost kept only as a competitor. And the decision threshold is a co-optimized variable, which has no counterpart in classification.

Several papers motivate the hardware side. DRACO shows that on systolic arrays, memory-bound layers can run faster with more arithmetic. This "latency paradox" means that operation counts, and by extension bit-width alone, can mis-rank configurations. Partition predictors are full of small, memory-bound convolutions, so the paradox is relevant here. Air Learning shows that injecting measured hardware latency into training closes a 37.73% flight-time gap to 0.5%. We borrow that measure-then-train logic. The endoscopic hardware-aware NAS work translates power and temperature limits into constraints on a ZCU102 FPGA for real-time video. It never quantizes and never touches a codec.

Two papers constrain the design. The hardware-vs-software power study found that CPU software energy counters correlate with a wall meter at 0.98 to 0.99 during H.264/HEVC encoding. That means board measurement must earn its cost on the accelerator, not on the CPU. Walker et al. show that simulator power models fail on unseen workloads and give a recipe for correcting them from measurements. We use that recipe to build the "corrected simulator" comparison below.

The low-bit quantization methods QuIP and CRVQ protect outlier channels with rotations and extended codebooks. They are built for language-model weights and lack native low-bit kernels; QuIP runs 1.5× slower than its predecessor at inference. We treat salient-channel protection as an optional ingredient, to be tested only if the sub-4-bit regime shows the outlier-driven collapse these papers describe.

Research Program

The first stage is a diagnostic. We quantize both predictors with standard post-training uniform quantization at 8, 6, 4, 3 and 2 bits. We log every CTU decision against the floating-point decision and against the exhaustive search. Each flip is classified as skip-error (rate loss) or keep-error (time loss) and indexed by block size, tree depth, quantization parameter and distance from the threshold. This stage alone answers an open question: where along the precision axis do these predictors fall off a cliff, and is the cliff in decisions or in raw outputs?

The second stage builds the deployment. Quantized predictors are compiled to the ZCU102's programmable logic, chosen because the endoscopic work characterized it for real-time video. The rate-distortion search the predictor leaves undone is counted in VTM, the reference encoder, as a work surrogate for a hardware search engine. That search work is converted to time using a per-operation cost measured on the board's processors. This is a stated approximation: we do not build a full hardware VVC encoder. Per-CTU predictor latency and board power come from on-board sensors, using repeated runs until a Student's-t confidence threshold is met, following the power study's protocol.

The third stage is the optimizer. Bit-widths are calibrated with DeepQTMT's rate-distortion-aware loss, extended so that a flip toward skipping a split is penalized by its measured bitrate cost. The pair $(b, Th)$ is searched against the constraint above using measured latency. Optional activation-salience protection is added only where the diagnostic shows outlier channels driving flips.

Baselines

The naive reference point is the unaccelerated VTM encoder, which defines zero BD-rate loss and the full search cost. Alongside it we run each predictor in floating point at its published threshold, which must reproduce the reported 44.65% to 66.88% (DeepQTMT) and 17% to 30% (MBMP-CNN) time savings before any quantized comparison is trusted. Both run on the same sequences and configurations as every other arm.

The "do nothing clever" quantized baseline is uniform 8-bit post-training quantization with the original threshold. It is what an engineer would ship first. If our method cannot beat it at 8 bits, the interesting regime is lower precision, and we say so.

The strongest existing method is HAQ-style mixed-precision search. We run it with the same search budget, measured in number of evaluated configurations, and the same calibration data. Its latency comes from an analytical FPGA cost model, and its reward is predictor agreement with exhaustive-search labels. HAQ reports no number on video encoding, and no reported result was found on Lacuna for quantized partition predictors at all. The comparable figure is therefore only its classification result: 1.4× to 1.95× latency reduction versus fixed 8-bit. Beating it means lower BD-rate at an equal measured deadline-miss rate, or a lower miss rate at equal BD-rate, on held-out 4K sequences.

A second arm gives HAQ the correction from Walker et al.: its simulator latency is refit to board measurements. This tests whether a corrected simulator, rather than in-the-loop measurement, is enough.

The ablation that isolates our key ingredient is sequential tuning. We take the best bit-width assignment found without threshold co-optimization, then retune $Th$ on the validation set. A second ablation drops the asymmetric flip penalty.

Why not just…?

Why not just quantize to INT8 and ignore it, since inference is 0.21% of runtime? In software that argument is correct, and we concede it: the time savings in MBMP-CNN come from skipped search, not from fast inference. The argument stops holding once the search itself is accelerated in hardware. At that point the predictor shares a roughly 66 µs per-CTU budget with that hardware, and at 4K its activation traffic competes for the same memory. The low-delay benchmark (Teng et al.) excluded UHD content because neural models hit exactly these memory limits. Stage two measures the predictor's share of the budget directly. If 8-bit inference fits comfortably with flips below one percent, the honest conclusion is that precision is not the bottleneck, and we report that.

Why not just retune the threshold after quantizing? This is the real threat. MBMP-CNN already exposes $Th$ as a speed-quality knob, so a shifted probability distribution may simply be absorbed by moving it. The sequential-tuning ablation settles the question. Joint selection should win only if quantization noise is non-uniform across block sizes and depths, because a single global threshold cannot correct depth-dependent bias. The diagnostic in stage one shows in advance whether that non-uniformity exists. If it does not, we drop joint optimization and recommend per-depth thresholds over the post-hoc approach.

Why not just trust a simulator, or a corrected one? Software energy counters tracked a hardware meter at 0.98 to 0.99 on CPU encoders, so measurement on the board may be unnecessary. The test is whether simulated and measured latency rank our candidate configurations the same way, as scored by Kendall rank correlation. DRACO's latency paradox predicts disagreement for memory-bound layers. If the corrected simulator achieves high rank agreement, it replaces the board in the loop, and the contribution narrows to the decision-aware objective.

Proposed Experiments

The intra predictor is trained and calibrated on the CPIV database, roughly one billion CUs. The inter predictor is trained on BVI-DVC and YouTube-UGC, following MBMP-CNN. Held-out evaluation uses the nine 4K sources of BVI-CC. That benchmark reports VTM at 7× to 9× slower than HEVC, and VMAF at an SROCC of 0.84 to 0.87, against roughly 0.55 to 0.60 for PSNR. We therefore report BD-rate in both PSNR and VMAF. Inter experiments run under Random Access, to match MBMP-CNN, and under Low-Delay B, the real-time setting.

Metrics are reported per arm and per bit-width. They are BD-rate, search work saved, the flip rate split into skip and keep errors, the measured deadline-miss rate at $\tau$, and board power. We also report Kendall rank correlation between simulated and measured latency over the full candidate set.

Three results would decide the claim. The method should dominate both HAQ arms on the BD-rate versus miss-rate frontier at 4 bits and below. It should beat sequential threshold retuning by a margin larger than run-to-run variance, estimated from repeated encodes. And skip-side flips should account for most of the BD-rate loss in the plain quantization arms while the asymmetric loss removes them. If all three hold, quantization loss for encoder-control networks is a decision problem that must be measured on the device. If only the third holds, the contribution reduces to a diagnostic and a calibration loss, which is still useful but smaller.