Measuring the Universality Penalty on Real Images

A learned image codec has two parts: an encoder that turns an image into a short bitstream, and a decoder that turns the bitstream back into an image. Decoders can be tuned either for low pixel error, which gives blurry but faithful images, or for realism, which gives sharp images whose fine detail is partly invented. A line of information theory argues that one encoder, trained only to minimize squared error, is enough for every point on this spectrum. Only the decoder needs to change, and the claimed cost in extra bits is zero. That result, called encoder universality, is proven only for Gaussian sources. It has been checked mostly on handwritten digits.

This proposal measures how many extra bits universality actually costs on natural photographs. The cost is separated from the noise of imperfect training. We then ask whether cheap synthetic sources, calibrated to image statistics, predict that cost. If they do, synthetic validation in this field is legitimate when done carefully. If they do not, the field has been validating a real-world claim on data that cannot reveal its failure.

Core Claim

We conjecture that the zero-rate-penalty result is a property of Gaussian sources and does not carry over to natural images. On real images we expect a positive penalty that is concentrated at low bitrates and high realism targets. We expect it to grow with how strongly the source departs from Gaussian statistics, with heavy tails and sparse edges and text being the main departures. An encoder trained only for squared error spends its few bits on high-energy structure. It drops low-energy but perceptually decisive content such as thin strokes, text, and small faces. No decoder can recover what the bitstream never carried. It can only invent it, and that matches the hallucinations reported by High-Fidelity Generative Image Compression and Rethinking Lossy Compression.

What would be new is a controlled estimate of this penalty, separated from training noise, with confidence intervals, on Kodak, CLIC2020 and DIV2K. We would also test whether a ladder of synthetic sources predicts it.

Where the Evidence Stops

The theory comes from three papers, all proven for Gaussian sources under squared error and the Wasserstein-2 distance (a measure of how far apart two distributions are).

- Universal Rate-Distortion-Perception Representations proves the zero penalty.

The empirical support is thin.

  • The first paper tests only MNIST and SVHN and reports "very close" performance with no numbers in our file.
  • The classification paper tests only MNIST. It also finds that universality breaks when the encoder is trained at high distortion. That finding motivates our hypothesis.
  • On Perceptual Lossy Compression supplies the two-step method we reuse: train the encoder for squared error, freeze it, then train a conditional adversarial decoder. It validates only on MNIST.
  • Optimally Controllable Perceptual Lossy Compression is the closest prior work. It runs a shared squared-error encoder with two decoders on Kodak and reports a better Perception Index than HiFiC. It does not compare against end-to-end specialized codecs of matched architecture, it does not report rate penalties, and our file contains no numbers from it.
  • Optimal Neural Compressors for the RDP Tradeoff shows that perceptual constraints at low rate need shared random dither, meaning random noise known to both sender and receiver. It stops at Gaussian vectors, MNIST and speech. This constrains our design: quantization type must be treated as an experimental factor.

The synthetic-data literature supplies the evaluation lens.

We carry both ideas into compression. A synthetic source counts as valid for universality claims only if it reproduces the shape of the penalty surface seen on real images.

Research Program

Definition of the penalty. For a target realism level $P$, we define it as

$$\Delta R(P) = \text{BD-rate}\big(R_{\text{univ}}(D, P),\; R_{\text{spec}}(D, P)\big).$$

  • $R_{\text{univ}}$ is the bits per pixel needed by the frozen squared-error encoder with a retrained decoder.
  • $R_{\text{spec}}$ is the same quantity for a codec trained end to end for that target.
  • BD-rate is the standard average percentage bitrate difference between two rate-distortion curves.

On images, realism is measured with FID and KID. FID is the Wasserstein-2 distance between Gaussian fits of deep image features, so it is the closest image-scale stand-in for the theory's metric. LPIPS is reported as a secondary metric.

Separating training noise. A positive $\Delta R$ could just mean that adversarial decoders are hard to train. So every specialized codec also gets a "recipe floor" control. We freeze its encoder and retrain its decoder from scratch with exactly the universal decoder's recipe. The penalty attributable to the encoder is the universal model's gap over this floor, not its gap over the original specialized model.

Shared design. All models share one hyperprior backbone, the architecture family behind both HiFiC and the Mean-Scale Hyperprior. All are trained on the same data and for the same number of steps per operating point. All operate at three bitrate targets and four realism weights.

The synthetic ladder. We run the identical protocol on five sources of increasing realism:

  1. Gaussian vectors, where the theory predicts zero penalty and the exact Wasserstein-2 distance can be computed. This rung checks the pipeline.
  2. Gaussian scale mixtures fitted to the wavelet statistics of the training photographs, which add heavy tails.
  3. Procedurally generated occlusion images with text overlays, which add sparse edges and glyphs.
  4. MNIST and SVHN, the sources the prior validations used.
  5. Kodak, CLIC2020 and DIV2K.

The test is whether the penalty surface over rate and realism on each synthetic rung has the same shape as on real photographs. We measure shape match by rank correlation across the twelve operating points, plus a ratio of magnitudes. Adding statistical properties one rung at a time also tells us which property drives the penalty.

Machine-vision axis. Universality claims also cover downstream tasks. SC2 Benchmark shows that task-supervised encoders deliberately discard task-irrelevant content. Using sc2bench, we attach COCO detection and ImageNet classification heads to the frozen universal encoder. We measure the rate penalty against that benchmark's supervised encoders. We also measure how much accuracy drops when a detector runs on realism-decoded images versus squared-error-decoded images. That drop counts the cost of hallucination in task terms.

Baselines

Classical codecs. The do-nothing reference is BPG and JPEG input compression followed by the same evaluation, run through identical metric code on identical images. It anchors the bitrate axis. HiFiC reports that it was preferred by humans at 0.237 bpp over BPG at 0.504 bpp, and matched BPG at 0.390 bpp while using 0.120 bpp (High-Fidelity Generative Image Compression). A universal model that does not clearly beat BPG at realism-matched points has failed outright.

Specialized end-to-end codecs. The strongest competitor is HiFiC-style training: one codec per operating point, trained jointly with distortion and adversarial losses. This is exactly the design universality claims to make unnecessary. The human-preference numbers above are the only reported figures in our file; our file has no reported FID-at-bpp result for it on Kodak. We will retrain it within our backbone, with the same tuning budget per point as the universal decoders and the same training data. Beating it means a universal BD-rate penalty, measured against the recipe floor, whose 95% bootstrap interval includes zero at a given operating point. That would be evidence that universality holds there.

Prior shared-encoder system. Optimally Controllable Perceptual Lossy Compression runs one squared-error encoder with an error-minimizing decoder and a realism decoder, and blends their outputs linearly. No reported result was found on Lacuna for its FID or rate on Kodak. We reimplement it in our backbone and treat it as one realization of the universal arm.

Diffusion decoding. Training-Free RDP Traversal With Diffusion uses pre-trained diffusion models, which generate images by iteratively removing noise, to move across the tradeoff without any training. No reported result was found on Lacuna for its numerical LPIPS or FID on Kodak or DIV2K. We report it with its decoding time per image.

Supervised task encoders. For the machine-vision axis, the baseline is the Entropic Student from SC2. No reported result was found on Lacuna for its detection accuracy.

Ablations isolating the encoder's training target. The first ablation is the recipe floor already described. The second adds a small realism term to the encoder's loss, giving a "near-universal" encoder. The third swaps deterministic quantization for shared dither, so we can see how much of the penalty is a quantization artifact.

Why not just…?

Why not just cite the existing Kodak results for shared-encoder systems? Optimally Controllable Perceptual Lossy Compression compares against HiFiC, which uses a different architecture and different training. It reports Perception Index rather than rate penalty. A favorable comparison of that kind cannot separate "universality is free" from "this decoder is better engineered." Our matched backbone and recipe floor exist to make that separation.

Why not just trust the MNIST and SVHN validations? Whether toy data reveals the penalty is exactly what the synthetic ladder tests. There is already a warning in the file. Universal Rate-Distortion-Classification Representations found universality failing under high-distortion encoders even on MNIST. On the Equivalency, Substitutability, and Flexibility of Synthetic Data found that structurally simple synthetic data actively hurt transfer.

Why not just use training-free diffusion decoding and skip encoder questions? Its optimality proof assumes Gaussian sources, and it works in a latent space that departs from the theory. Its decoding needs many network evaluations and is orders of magnitude slower than a single-pass decoder, according to the paper itself. We include it as a baseline and report its time per image rather than assume it settles the question.

Why not just add a small realism term to the encoder? This is a real threat. If a near-universal encoder removes most of the penalty at little cost in squared error, the practical lesson becomes "slightly perceptual encoders are universal enough," and strict universality matters less. The ablation measures exactly this trade. We will report it as a headline result if it wins.

Proposed Experiments and Decisive Checks

Check 1: the Gaussian rung. It must reproduce zero penalty within its confidence interval. If it does not, the pipeline is broken and nothing downstream is interpreted.

Check 2: natural images. On Kodak, CLIC2020 and DIV2K, the hypothesis predicts a penalty above the recipe floor that is positive and grows as bitrate falls and the realism target tightens. The hypothesis is refuted if the penalty's interval includes zero at the lowest bitrate and highest realism target.

Check 3: the synthetic ladder. The test is the rank correlation between each rung's penalty surface and the natural-image surface. We expect MNIST and SVHN to correlate weakly and the glyph-bearing procedural rung to correlate strongly. If that ordering holds, synthetic validation is legitimate only when the synthetic source carries sparse, low-energy structure.

Check 4: machine vision. On COCO, we report detection mean average precision (the standard COCO detection score) against bitrate for three systems: universal heads, Entropic Student, and BPG input compression. Separately, we report the accuracy drop caused by realism decoding. A large drop at matched FID would show that the realism the theory optimizes does not protect downstream machine tasks.

Check 5: video, if time allows. The protocol extends to video using URDP_flow on UVG.