Some neural video codecs now fine-tune themselves on the very video they are compressing. The encoder slightly retrains the decoder network on that one clip and sends the small weight change along with the compressed frames. This buys 20–27% bitrate savings. Every one of these systems measures quality the same way: squared pixel error, averaged uniformly over the frame. Under a tight budget for the weight update, the optimizer therefore spends it on whatever produces the most pixel error. Often that is rippling water, foliage, or sensor noise, which viewers barely look at. Meanwhile, decades of eye-tracking research show that where people look is governed by semantic content such as faces, text, and moving agents, not by pixel contrast.
This note proposes a narrow change and a careful test of whether it matters. We weight the fine-tuning objective, and the choice of which image patches the encoder trains on, by a predicted gaze map. We then check whether this improves quality where humans actually look, at equal total bitrate. The key question is not whether saliency helps compression; existing region-of-interest codecs already show it does. The question is whether saliency still helps inside the per-video fine-tuning loop once the ordinary compressed-frame bits have already been steered by saliency. If it does not, the right answer is to combine two existing methods, and we will say so.
Core Claim
In instance-adaptive compression, the transmitted model update is a global resource. A small number of weight changes apply to every pixel of every frame. We claim that telling the adaptation which pixels matter changes what those changes encode. It shifts them away from fitting background texture and toward reproducing the fine structure of semantic regions: skin, eyes, glyph edges, hands. This gain should be additive to, not redundant with, spatially steering the bits of the compressed frames themselves. A second, practical claim is that the same gaze map can cut encoder-side training cost. If the encoder trains only on patches sampled in proportion to predicted gaze, it backpropagates through a fraction of the video. That addresses the encoding-latency problem that currently limits instance adaptation to offline use.
Grounding in Prior Work
The adaptation machinery comes from three papers. Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set fine-tunes a Scale-Space Flow codec (SSF, a neural codec that predicts each frame from the previous one using a blurred motion field) on the test sequence. It codes the weight change under a spike-and-slab prior, which pushes most of the change to exactly zero. It reports 21–27% BD-rate savings over SSF, updating only about 0.8% of parameters. BD-rate is the average bitrate change at equal quality across a rate-distortion curve. Parameter-Efficient Instance-Adaptive Neural Video Compression replaces full fine-tuning with low-rank adapters on convolutions. It shows that adapting the decoder alone is enough and cuts the update to 43 kB from 8,522 kB. Group-aware Parameter-efficient Updating for Content-Adaptive Neural Video Compression trains adapters over whole groups of frames using spatial patches, which reduces error drift. All three optimize uniform squared error and sample patches uniformly. We reuse their mechanisms unchanged and alter only the objective and the sampling.
The case that gaze is predictable and semantic comes from the saliency literature. Using Deep Features to Predict Where People Look shows that frozen ImageNet features capture 88% of the explainable information in fixation data. Predicting Visual Fixations shows that low-level contrast explains little by comparison. For video we use TASED-Net, a 3D convolutional saliency predictor over roughly one-second windows that reports a Normalized Scanpath Saliency of 2.667 on DHF1K. Normalized Scanpath Saliency is the average normalized saliency value at human fixation points. TASED-Net runs only at the encoder, so its 82 MB footprint and 0.06 s per frame do not burden the decoder.
Two papers constrain the design. Region-of-Interest Based Neural Video Compression steers latent bits with a mask-driven gain network on a static SSF. It reports a 69.3% BD-rate gain inside the region on DAVIS, but also boundary artifacts and flicker from binary masks. FOVQA finds that how fast quality falls off away from the fixation point drives viewer annoyance. Together they rule out hard masks. Our weights will be continuous, spatially blurred, temporally smoothed, and floored above zero.
The closest adaptive work, Towards High-Quality and Efficient Video Super-Resolution via Spatial-Temporal Data Overfitting, also changes what an overfitted model trains on. It clusters patches by a base model's PSNR error and oversamples hard patches, which by construction favors high-error texture. We replace error-driven selection with gaze-driven selection, and we work in a compression codec rather than a super-resolution upscaler. Saliency Driven Perceptual Image Compression weights a globally trained image codec by object saliency. It has no temporal model and no per-instance adaptation.
Research Program
The system is an SSF backbone with decoder-only convolutional low-rank adapters, following the ACCV design, with updates coded under a spike-and-slab prior. We choose SSF because the region-of-interest latent-scaling codec is built on the same backbone, which lets us combine the two cleanly. For each training frame $t$, TASED-Net produces a saliency map $s_t$. We smooth it spatially and across time and convert it to a weight map $w_t = \epsilon + (1-\epsilon)\,\tilde{s}_t$, normalized to mean one. Here $\tilde{s}_t$ is the smoothed map rescaled to $[0,1]$ and $\epsilon$ is a floor that keeps the periphery from collapsing. The adaptation then minimizes
$$\mathcal{L}(\delta) = \mathcal{R}(y) + \mathcal{R}(\delta) + \lambda \sum_{t}\sum_{i,j} w_t(i,j)\,\lVert x_t(i,j) - \hat{x}_t(i,j;\delta) \rVert^2 ,$$
where $\delta$ is the adapter update, $\mathcal{R}(y)$ is the bitrate of the compressed frame latents, $\mathcal{R}(\delta)$ is the bitrate of the coded update, $x_t$ and $\hat{x}_t$ are original and reconstructed frames, and $\lambda$ sets the rate-quality tradeoff. The mean-one normalization keeps $\lambda$ comparable to the uniform baseline. Separately, we replace uniform patch sampling with sampling proportional to patch-averaged $w_t$. Crucially, we run this sampling at reduced patch budgets (25%, 50%, 100% of patches) to measure the encoding-compute savings.
Three analyses accompany the rate-distortion results. First, we attribute where the update's effect lands. For each method we measure the per-pixel reduction in error due to $\delta$ and correlate it with human fixation density. This directly tests the mechanism, namely that uniform adaptation spends its update on background. Second, we stratify results by the entropy of human fixations per clip. Scenes with scattered attention, such as crowds or sports, should show little benefit, and a method that helped equally everywhere would suggest it is not really about gaze. Third, we sweep $\epsilon$ and the smoothing width. We track peripheral quality and frame-to-frame flicker in the region boundary to find where FOVQA's fall-off penalty bites.
Baselines
The naive reference point is the frozen, globally trained SSF with no adaptation and no saliency. It anchors every curve. Classical encoders enter as x265 with and without saliency-driven delta-QP, a per-block quantization offset derived from the same TASED-Net maps. This is the cheapest way any practitioner would use a gaze map. We encode it with matched rate targets and the same maps. It counts as beaten only if we win in fixation-weighted quality at equal bitrate.
The strongest uniform adaptive competitor is decoder-only convolutional low-rank adaptation from Parameter-Efficient Instance-Adaptive Neural Video Compression, which reports −19.29% average BD-rate against x265. We also include full-model InstA from Instance-Adaptive Video Compression, which reports 21–27% BD-rate savings over SSF. We reimplement both on our SSF with the same adapter rank, optimizer, step count, and learning-rate search per clip. The only difference from our method is the weighting and sampling. The uniform-sampling group-of-frames scheme of Group-aware Parameter-efficient Updating, reported at 2.25% over DCVC_DC, sits on a different backbone. We therefore port only its group-of-frames patch training as a uniform-sampling variant rather than claim its numbers.
The strongest saliency competitor is Latent-Scaling ROI SSF from Region-of-Interest Based Neural Video Compression, reported at a 69.3% BD-rate gain inside the region on DAVIS. We feed it our continuous TASED-Net maps rather than binary masks, so it gets the same information. The most important competitor is the composition of existing methods: ROI SSF latent scaling plus uniform low-rank instance adaptation. We make it the reference to beat.
The ablations form a two-by-two factorial: saliency in latent allocation (off/on) crossed with saliency in the adaptation objective (off/on). A further axis separates weighted loss from weighted sampling. An STDO-style arm samples by base-model PSNR error instead of gaze, which isolates "semantic" from "hard." A shuffled-saliency arm uses gaze maps from a different clip, which controls for the effect of merely reweighting.
For every learned method, the total bits counted include latents plus the coded update. We have found no reported result on Lacuna for any instance-adaptive codec under fixation-weighted metrics, so all numbers in that regime come from our own runs. Beating the composition means a statistically reliable BD-rate reduction in fixation-weighted PSNR over the ROI-plus-uniform-adaptation arm, across clips, at no more than a small uniform-PSNR loss.
Why not just…?
Why not just steer the latent bits with saliency and adapt the weights uniformly? This is the real threat. The region-of-interest codec already moves bits toward salient regions, and once those regions receive more latent bits, uniform adaptation may naturally see less error there and redirect itself. The factorial design settles it. If the "saliency in adaptation" factor adds nothing on top of latent scaling in fixation-weighted BD-rate, we will report that the combination of existing methods suffices. We will then recast the contribution as the encoding-cost reduction from gaze-proportional sampling, which latent scaling cannot provide.
Why not just use saliency-driven delta-QP in x265 or VVC? It is nearly free and works with deployed decoders. The evidence against it as a ceiling is indirect. Uniform instance adaptation already beats H.265 by 5–44% BD-rate per Instance-Adaptive Video Compression, and quantization offsets cannot learn clip-specific textures. We run it with identical maps. If it matches us in fixation-weighted quality, the learned approach is not worth its cost for this goal.
Why not just oversample hard patches, as STDO does? Error-driven sampling may already capture much of the benefit, since faces and text are sometimes hard. The PSNR-error arm tests this. We expect its update-attribution map to correlate with texture energy rather than fixation density, and its gains to concentrate in low-fixation regions. If instead it matches gaze sampling, semantics adds nothing beyond difficulty.
Why not just train one global codec with a saliency-weighted loss? This removes per-clip encoding cost. However, Instance-Adaptive Video Compression shows that small adapted decoders beat much larger global ones, and domain shift such as animation limits global models. We include a globally saliency-fine-tuned SSF so the per-clip benefit is measured, not assumed.
Proposed Experiments
The primary evaluation uses DHF1K, because it supplies human fixations per frame. We evaluate with human fixation maps, blurred to about one degree of visual angle, while training uses only TASED-Net predictions. This avoids grading the method with the same predictor that guided it. The metrics are fixation-weighted PSNR and MS-SSIM (multi-scale structural similarity), reported as BD-rate over total bits. DHF1K sources are not pristine, so we filter to higher-quality clips. We treat the gains as relative comparisons among codecs on identical inputs, not as absolute fidelity. LIVE-FBT-FCVR, which also carries gaze, serves as a second fixation-grounded test. UVG, HEVC Class B/C, and MCL-JCV, with predicted saliency only, measure the uniform-PSNR cost relative to published adaptive baselines. Big Buck Bunny probes whether gaze weighting survives domain shift. Encoding cost is reported as wall-clock and backpropagated patches per clip.
We also run a small paired-preference study at matched bitrate on a DHF1K subset. It compares our method against the ROI-plus-uniform-adaptation arm, because metric gains in weighted PSNR are not by themselves evidence of better viewing.
The decisive checks are three. First, the interaction cell of the factorial, which shows whether saliency in adaptation helps beyond saliency in latents. Second, the update-attribution correlation, which shows whether the update really moves toward fixated pixels. Third, the encoding-budget sweep, which shows whether gaze-proportional sampling at 25% of patches retains most of the fixation-weighted gain of full uniform adaptation. A null result on the first check combined with a positive result on the third would still be useful. It would redirect the claim to cheaper encoding rather than better allocation.
References
- Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set
- Parameter-Efficient Instance-Adaptive Neural Video Compression
- Group-aware Parameter-efficient Updating for Content-Adaptive Neural Video Compression
- Using Deep Features to Predict Where People Look
- Predicting Visual Fixations
- TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection
- Region-of-Interest Based Neural Video Compression
- FOVQA: Blind Foveated Video Quality Assessment
- Towards High-Quality and Efficient Video Super-Resolution via Spatial-Temporal Data Overfitting
- Saliency Driven Perceptual Image Compression