A learned image codec compresses a picture with a neural encoder, rounds the result to integers, and writes those integers with an arithmetic coder guided by a learned probability model. When such a codec is adapted to an unusual image at encoding time, for example a screenshot, a piece of vector art, or a pathology slide, the adaptation is almost always done by gradient descent on a smooth stand-in for the real objective. That stand-in replaces rounding with added noise, replaces the true file size with the model's estimate of it, and runs a floating-point decoder instead of the integer decoder that is actually deployed. This note proposes a different split of the work. The stand-in's gradient is kept as a cheap guide. A small black-box optimizer then measures the real objective, meaning actual bytes written plus distortion from the deployed decoder, and learns only the correction between the guide and the truth. The bet is that this correction is small and low-dimensional on ordinary photos but large on out-of-distribution content, which is exactly where current test-time adaptation goes wrong.
The motivating evidence comes from Test-time Adaptation for Image Compression with Distribution Regularization. That paper found that gradient-based refinement on out-of-domain images can improve distortion while making files larger. On one screen-content set, vanilla refinement increased bitrate, because the latents drifted to places where the probability model's estimate stops being trustworthy. Their fix is a penalty that keeps latents near familiar territory. Ours is to stop trusting the estimate at all and instead measure the quantity the codec is actually judged on.
Core Claim
The claim has two parts. First, under domain shift, the gap between the differentiable training objective and the real rate-distortion cost grows. Here "rate" means the true coded length, including the bits needed to send any adapted parameters, and "distortion" is measured through the deployed quantized decoder. Second, if the real cost is modeled as a scaled copy of the stand-in plus a residual, the residual can be learned from a few dozen forward-only evaluations. Learning only the residual, instead of the whole function, is what lets black-box optimization work in the hundreds-to-thousands of dimensions that lightweight codec adapters occupy. Without the stand-in as a guide, black-box search would run into the curse of dimensionality: a flat, uninformative surrogate, as described in Frank Hutter and Papenmeier et al..
Concretely, let $\theta$ be the quantized adapter parameters sent with the image. Let $f(\theta)$ be the true cost, defined as the actual bitstream length for latents plus adapter, plus $\lambda$ times the distortion of the integer decoder's output. Let $g(\theta)$ be the usual relaxed, differentiable estimate. We model
$$f(\theta) = \alpha\, g(\theta) + r(\theta), \qquad r \sim \mathcal{GP}(0, k),$$
where $\alpha$ is a learned scale and $r$ is a residual with a Gaussian process prior. A Gaussian process is a probabilistic regression model that gives both a prediction and an uncertainty. Because derivatives of a Gaussian process are again Gaussian processes, the posterior over the true gradient is $\alpha \nabla g(\theta) + \nabla r(\theta)$. One backward pass gives the first term. The black-box queries only need to pin down the second term.
Research Program
The method supplies two things. The first is the codec and adapter design from Universal Deep Image Compression via Content-Adaptive Optimization with Adapters, which inserts 768 decoder adapter parameters and adapts them per image by backpropagation. The second is the local gradient-estimation idea of GIBO from Local Policy Search with Bayesian Optimization. GIBO keeps a Gaussian process over a black-box objective and picks query points that most reduce uncertainty about the gradient at the current iterate, then takes a step. GIBO learned the full gradient from scratch and was shown only up to 36 dimensions, in control tasks. We change it in two ways. We give it the stand-in gradient as a prior mean. We also confine the residual's gradient to a small subspace, of dimension $k$ between 16 and 64, spanned by recent stand-in gradients plus a few random directions. This subspace trick echoes the low-effective-dimensionality arguments behind Krause et al. and Letham et al., but here it is applied only to the correction, not to the whole search. A trust region in the style of TuRBO (Eriksson et al.) governs the step size. It grows when true-cost evaluations confirm improvement and shrinks otherwise, and it keeps the Gaussian process fitted on a small local window so that fitting cost, which grows with the cube of the number of points, stays negligible.
Each true-cost query is a forward pass, a real arithmetic-coding pass, and an integer decode. Because the adapter values are evaluated after quantization, the straight-through gradient approximation relied on by Overfitting for Fun and Profit and Instance-Adaptive Video Compression is never needed for the objective itself. Those papers fine-tune whole models by backpropagation and send quantized weight updates. We keep their idea of charging the model-update bits to the rate, but measure those bits exactly.
The first deliverable is diagnostic rather than algorithmic. We will measure the stand-in-to-truth gap on in-domain and shifted images, both as the error in cost and as the angle between stand-in and true gradients. The true gradients will be estimated by dense finite differences on a small sample. If the gap does not grow with shift, the method has no reason to exist and we will report that.
Baselines
The naive reference is the unadapted pretrained codec, with the same backbone, the same encoder latents, and no per-image optimization, measured against VTM (H.266/VVC).
The strongest competitor on the same parameters is gradient adaptation of the same adapters. This is Adam on the relaxed stand-in with straight-through quantization, as in Universal Deep Image Compression via Content-Adaptive Optimization with Adapters. That paper reports a 2.87% BD-rate improvement over VVC on vector art, where the unadapted deep model was 25.3% worse than VVC, at about 260 seconds per image on a V100. BD-rate is the average bitrate saving at equal quality across a rate curve.
The strongest competitor on the shift problem overall is distribution-regularized latent refinement from Test-time Adaptation for Image Compression with Distribution Regularization. It reports an average 27.00% BD-rate saving across cross-domain sets on the AR-CM backbone, against 21.41% for baseline latent refinement. Because that method adapts latents while ours adapts adapters, we will run it as a baseline in its own right. We will also run it as a shared first stage, applying it to the latents in every adapter arm so that differences come only from how the adapters are optimized. Beating it means a lower true-bit BD-rate on the same backbone and datasets, or equal BD-rate at clearly lower encode compute.
The derivative-free competitor is CMA-ES, an evolutionary search that adapts a sampling covariance. Following PACE from Subspace Optimization for Backpropagation-Free Continual Test-Time Adaptation, it will run in a low-dimensional subspace over the same adapter parameters and on the same true cost. That work reports gains of 1.7 to 4.0 points over other backpropagation-free methods for classification. No reported result was found on Lacuna for CMA-ES or any black-box optimizer on rate-distortion objectives.
We also run TuRBO on the true cost with no stand-in, and GIBO with no stand-in prior.
Three ablations isolate the ingredients. The first is stand-in gradient steps whose acceptance and step size are decided by true-cost evaluation, with no residual model. The second is our method with $\alpha$ fixed to 1. The third is our method with the full-dimensional residual instead of the subspace.
All arms receive the same budget. We count it in forward-pass equivalents, with a backward pass charged as roughly two forwards. We also report wall-clock time on the same GPU. Every arm gets the same number of tuning trials on a held-out development split per domain.
Why not just…?
Why not just run Adam on the stand-in, as everyone does? If the stand-in tracks the true cost, Adam wins, because gradients are cheaper than black-box queries. That is why our in-domain Kodak runs are expected to show no gain and serve as a control. The case against Adam out of domain rests on the bitrate inflation documented by Chen et al. on SCID and on the straight-through mismatch for quantized parameters. Whether these effects actually cost BD-rate on true bits at matched compute is precisely what the gap diagnostic and the main comparison settle.
Why not just use the stand-in gradient and check each step against the true cost? This is the most serious threat. It captures most of the benefit, namely never accepting a step that truly hurts, with no Gaussian process at all. It is the first ablation. If it matches our method, the honest conclusion is that verification, not residual modeling, is what matters. The residual model earns its place only if the true gradient points away from the stand-in gradient, not merely if the stand-in has the wrong magnitude. The gradient-angle diagnostic measures exactly this.
Why not just use CMA-ES in a subspace, as PACE does? CMA-ES handles black-box and quantized objectives and is well tested at this scale. Its weakness is that it ignores the free gradient information a differentiable stand-in provides, so it should need many more queries. If it matches us at equal budget on screen content and vector art, the stand-in prior is unnecessary, and we would recommend the simpler method.
Why not adapt offline on a few target images instead? Few-Shot Domain Adaptation for Learned Image Compression reaches about 27.8% BD-rate reduction on pixel art with around 25 target images. When such data exists, that is a strong option. Our method needs no target data. It can also be stacked on top of a few-shot adapted model, and we will test that combination on pixel art to see whether per-image correction still adds anything.
Proposed Experiments
In-domain control uses Kodak. Shifted evaluation uses the screen-content sets SCID, SIQAD, SCI1K, and CCT, the Behance Artistic Media vector art, comics, and line drawings, and pixel art and pathology from the few-shot suite. We use the backbones the baselines were reported on, AR-CM and Cheng2020, each with the adapter design of Tsubota et al. Rate is always the real arithmetic-coded length, including adapter bits. Distortion is PSNR and MS-SSIM from an integer-quantized decoder, and BD-rate is reported against both VTM and the unadapted model. For every arm we plot BD-rate against forward-pass-equivalent budget, from very small budgets up to the roughly 1,500-iteration regime reported by Campos et al..
Three results would be decisive. First, the gap diagnostic must show larger stand-in-to-truth error and gradient angle on shifted sets than on Kodak. Second, at matched budget on the shifted sets, our method must beat both the verified-stand-in ablation and subspace CMA-ES on true-bit BD-rate, while tying Adam on Kodak. Third, when the deployed decoder is integer-only, so that the stand-in is a floating-point approximation of it, the advantage should widen. If the first check fails, we publish the negative result. If the second fails only against the verified-stand-in ablation, we report that ablation as the practical method.
References
- Test-time Adaptation for Image Compression with Distribution Regularization
- Bayesian Optimization for More Automatic Machine Learning
- Understanding High-Dimensional Bayesian Optimization
- Universal Deep Image Compression via Content-Adaptive Optimization with Adapters
- Local policy search with Bayesian optimization
- High-Dimensional Gaussian Process Bandits
- Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimization
- Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
- Overfitting for Fun and Profit: Instance-Adaptive Data Compression
- Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set
- Subspace Optimization for Backpropagation-Free Continual Test-Time Adaptation
- Few-Shot Domain Adaptation for Learned Image Compression
- Content Adaptive Optimization for Neural Image Compression