Coding Overfitted Decoder Weights as Arithmetic on Past Images

Some of the most efficient neural image codecs do not ship one large network to everyone. Instead they train a tiny network for each image, and that network effectively is the compressed file. Because the decoder needs those exact weights, the weights have to travel inside every file. Examples are COOL-CHIC and C3, which use coordinate networks (small networks that map a pixel position to a colour). For these codecs the weight cost is a real share of the bitstream: up to roughly 20% in base COOL-CHIC, according to the research file. HyperCool shrinks this cost by predicting only a small correction to a shared base network. Even so, the correction is too expensive at low bitrates, and the codec then falls back to the unadapted base. Every existing system pays this cost from scratch for each image, even when the images arrive as a related collection: a photo album, a stack of scanned documents, a set of histology slides, or video frames.

This note proposes treating each image's weight correction as a vector. A new image's correction is coded as a quantized linear combination of corrections the decoder has already received, plus a small residual. The same additive structure that lets generative models edit images by vector arithmetic in their latent space (their compressed internal representation) is used here in the space of weight corrections, and the goal is purely to save bits. The predictors are the decoder's own previously decoded, already-quantized corrections, so encoder and decoder agree on them exactly. Synchronization is therefore free and needs no deterministic GPU training.

Core Claim

Suppose every image in a collection is overfitted as a small correction $\Delta w_i$ to the same frozen base decoder $w$. Then corrections of images from the same domain are close to linearly dependent. Concretely, the claim is that the following form will usually cost far fewer bits than coding $\Delta w_i$ independently:

$$\Delta w_i = \sum_{j \in \mathcal{P}_i} a_{ij}\,\widehat{\Delta w}_j + r_i .$$

Here $\widehat{\Delta w}_j$ is the quantized correction of an earlier image $j$, as the decoder holds it. $\mathcal{P}_i$ is a small set of predictor images chosen by the encoder. $a_{ij}$ are quantized scalar coefficients. $r_i$ is an entropy-coded residual. Only the indices in $\mathcal{P}_i$, the coefficients $a_{ij}$ and the residual $r_i$ are transmitted.

The practical consequence is that weight corrections become affordable in the low-bitrate regime where HyperCool currently has to drop them. The result should be a better rate-distortion curve over a whole collection than independent per-image coding, at unchanged decoding cost. The claim can fail in a specific, testable way. Latent-arithmetic work warns that additive offsets can leave the region where the generator behaves well, which is the "off-manifold" failure described in the file. Combined corrections might likewise land in parts of weight space that decode poorly. Testing this is the first experiment.

Where This Sits

The proposal takes its machinery from two codec papers. HyperCool predicts per-image corrections $\Delta w$ that are added to a frozen pre-trained decoder. It reports a 4.9% BD-rate reduction over non-overfitted Cool-chic on CLIC2020, and it shows that transmitting $\Delta w$ is cheaper than transmitting the full weights. However, it codes each image's $\Delta w$ independently. The proposal adds inter-image prediction on top of HyperCool and uses HyperCool's one-pass prediction as the starting point for optimization. COOL-CHIC and C3 supply the host codec: hierarchical latent grids plus a small synthesis network and a small entropy-model network, decoded at under 3k multiply-accumulate operations per pixel.

Two other papers motivate the idea and also constrain it. OSOA shows that an encoder and decoder can adapt a shared model online with no weight transmission, saving 47% of total space compared with fine-tuning and storing weights. The cost is bit-exact deterministic gradient updates on both sides, and the file notes this forces non-deterministic GPU operations to be disabled. This proposal achieves agreement differently. Its "shared state" is the set of integer-quantized corrections that were actually transmitted, so the decoder never runs a gradient step. Video Coding using Learned Latent GAN Compression codes differences between consecutive StyleGAN latent codes for face video. That is the closest use of vector arithmetic for compression, but it works in a frozen generator's latent space, applies only to faces, and does not count the generator's weight cost. Here the vectors are the transmitted weights themselves, the method is not tied to a domain, and the weight cost is counted in full.

Two further papers serve as context. Flexible Neural Image Compression via Code Editing avoids storing many models by editing latents against one fixed decoder; it removes model-count overhead but leaves the decoder unadapted. Compression via Pre-trained Transformers supplies the accounting principle used throughout this note, that total cost counts data plus model. It also gives a warning: model costs only amortize past a threshold. This is exactly why the collection-level comparisons below matter.

Research Program

The first stage is diagnostic. On a host codec built on HyperCool, overfit corrections independently for every image in several collections. Then measure how much of each $\Delta w_i$ can be explained by quantized combinations of the others, both per network module (upsampling, synthesis, entropy model) and per layer. Separately, decode with combined corrections and measure how distortion grows along straight lines between two images' corrections. This checks whether corrections that share a base network behave linearly, or whether combinations fall into poor regions of weight space.

The second stage builds the coder. For image $i$, the encoder greedily chooses up to $K$ predictors from the decoded history, with $K \le 4$ initially. It then optimizes the coefficients, the residual and the latent grids jointly under the objective

$$D(x_i, \hat x_i) + \lambda\big(R_{\text{latent}} + R(a_i) + R(r_i) + R(\mathcal{P}_i)\big).$$

Here $D$ is distortion and $R$ is the bit cost under the arithmetic coder. Quantization is handled by the soft-rounding scheme of C3. The residual's entropy model is conditioned on module and layer, and its scale shrinks as the history grows. The encoder also chooses the coding order of the collection. In a greedy most-similar-first ordering the cost of transmitting the order is a few bits per image, and it is counted.

The third stage adds two controls. One is a regularizer during overfitting that keeps corrections small and close to the history's span. This trades a little per-image fit for predictability, and an ablation will test whether the trade pays off. The other is a reset rule: if prediction does not save bits for an image, the encoder sends a flag and codes the correction independently. This rule guarantees the method never does worse than HyperCool by more than one flag bit per image.

Baselines

The naive reference is non-overfitted Cool-chic. It is a single shared coordinate decoder that transmits only latent grids, so its weight overhead is zero, and it is the anchor against which HyperCool reports its 4.9% CLIC2020 gain.

The strongest direct competitor is HyperCool with per-image corrections, both in one-pass form and with HyperCool-style fine-tuning. It addresses exactly the question of how to make per-image weight adaptation cheap, and it is the method this proposal extends. It will be run on the same base network, with the same latent grid configuration, the same number of encoding iterations per image, and the same λ grid. The only difference is whether $\Delta w_i$ is coded independently or predicted from history.

A fully overfitted Cool-chic/C3 encoding with independently transmitted weights marks the ceiling of what per-image adaptation can achieve. LotteryCodec is the strongest published low-complexity point on CLIC2020, at a −9.79% BD-rate reduction over VTM-19.1. It will be reported for context, though it transmits masks rather than corrections.

Two baselines target the "collection" framing directly. The first fine-tunes one shared correction on the entire collection, transmits it once, and divides its cost across the images. The second learns a fixed offline basis for corrections from a large training set, with coefficients coded per image. Both get the same encoding budget.

For the lossless variant, the reference is FNLIC. It reports 2.88 BPD on Kodak and 2.22 BPD on Doc24, with 2.30 BPD for JPEG-XL on Doc24. The file has no reported FNLIC numbers for Histo24 or LoveDA24, so those baselines will be measured rather than quoted.

The key ingredient is isolated by an ablation that keeps everything identical, including order choice and reset flags, but replaces the decoded-history predictors with random corrections of matched norm.

Beating the baselines means showing all three of the following on homogeneous collections. First, a negative BD-rate against independent HyperCool that is clearly outside run-to-run variance. Second, a gain that is concentrated at the lowest bitrates, where HyperCool falls back to no correction. Third, a strict improvement over the amortized single-shared-correction baseline on heterogeneous sets such as CLIC2020.

Why not just…?

Why not just fine-tune one decoder correction for the whole collection and send it once? This is the most serious threat. On a highly homogeneous set such as Doc24 it might capture most of the gain. However, it gives up per-image specialization. The file's evidence that fully overfitted codecs beat non-overfitted variants suggests that instance-specific structure matters, and the shared correction also requires the whole collection before any image can be sent. The comparison of the two baselines across homogeneous sets (Doc24, Histo24) and heterogeneous sets (CLIC2020, Kodak) settles this directly. If the shared correction matches the proposed method on homogeneous sets, the claim narrows to heterogeneous and streaming collections, and this note commits to reporting that outcome.

Why not just use OSOA-style online adaptation, which transmits no weights at all? OSOA requires the decoder to run bit-exact training steps, with the determinism penalties the file describes. That cuts against the main selling point of overfitted codecs, which is very cheap decoding. The proposed decoder only does integer multiply-adds over stored corrections. Decode time and a cross-hardware bit-exactness test will both be reported, alongside an OSOA-style variant that adapts the shared Cool-chic decoder online.

Why not just learn a better offline basis or entropy prior for the corrections? A static basis captures structure common to all images. It cannot capture what is specific to one collection, such as a particular scanner, stain or camera. This baseline is included at equal encoding budget. If it ties the history-based predictor, the conclusion is that corrections share a low-dimensional global structure rather than collection-specific structure. That result would still be useful, and it would redirect the work toward HyperCool-style priors.

Proposed Experiments and Decisive Checks

The lossy experiments run on CLIC2020 and Kodak, treated as heterogeneous collections, and on frames of UVG sequences, the video benchmark used by C3, treated as a strongly correlated collection. The lossless variant, built on FNLIC's image-specific network, runs on Doc24, Histo24 and LoveDA24. All rates count every transmitted bit: latents, coefficients, residuals, predictor indices, order and flags. This follows the accounting of Compression via Pre-trained Transformers.

Metrics are BD-rate against VTM and against HyperCool, PSNR, BPD for the lossless sets, decoding operations per pixel, and encode time. Each result is reported as a function of collection size, from 1 to the full set. With one image the method collapses to HyperCool; the useful question is how quickly the gain appears as images are added.

The first decisive check is the linearity diagnostic from the first stage of the research program. If quantized combinations of history explain little of each correction, or if combined corrections decode markedly worse than their endpoints, the core claim is false and the project stops there.

The second decisive check concerns the share of the bitstream spent on weights at the lowest λ. The method should keep corrections switched on in the regime where HyperCool turns them off, and the BD-rate gain against HyperCool should grow as bitrate falls. If instead the gain is flat across bitrates, the saving is coming from somewhere other than weight overhead.

The third check is the random-predictor ablation. It shows whether the decoded history, rather than the extra optimization freedom, is responsible for the gain.