Spend Realism Where It Is Cheap: Spatially Varying Perception–Distortion Control for Low-Light Image Compression

A night photograph is two images in one. The lit regions hold real, recoverable detail. The deep shadows hold almost nothing but sensor noise. Any codec that sends such a photo over a narrow channel and then cleans it up at the receiver has to decide two things for every region: how many bits to spend there, and how much the decoder may "invent" there. The decoder can either output a safe but blurry average, or a sharp, realistic-looking guess that may not match the true scene. Current systems make the second decision once for the whole image. That forces a bad bargain. A conservative setting blurs the lit regions, where sharpness would have been nearly free. An aggressive setting fabricates structure in the shadows, where the data cannot support it.

We propose a low-light codec in which the invention level, and the bit allocation trained jointly with it, vary pixel by pixel. Both are set by a single quantity the decoder can estimate: how uncertain the scene is in that region given what was transmitted. A one-line calculation turns that uncertainty into an explicit per-pixel rule for how much realism to allow. The central test is whether one such codec beats every globally tuned codec of the same architecture. Beating it means lower shadow error and better lit-region realism at the same bitrate, together with better human preference.

Core Claim

The perception–distortion tradeoff says a reconstruction cannot be both as close as possible to the truth on average and fully realistic. The Perception-Distortion Tradeoff proved this. Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff extended it to compression, where perfect realism costs at most a doubling of mean squared error over the distortion-optimal decoder. That bound is usually read globally. We read it locally.

Let $\mu(x)$ be the decoder's best average guess at pixel $x$, the "conditional mean". Let $s(x)$ be a sample drawn from the decoder's posterior, which is sharp and realistic. Let $m(x)$ be the local minimum achievable squared error, which is the posterior variance. Suppose the decoder outputs the blend $\hat{x} = \mu + \alpha(x)\,(s-\mu)$. Because the truth's deviation from $\mu$ and the sample's deviation from $\mu$ are independent and have the same spread, the expected local error is

$$\mathbb{E}\,\|x-\hat{x}\|^2 = \big(1+\alpha(x)^2\big)\, m(x).$$

At $\alpha=1$ this recovers the factor-of-two penalty. The formula says the price of realism is proportional to local uncertainty. In a well-lit region $m$ is tiny, so full realism is almost free. In a deep shadow $m$ is large, so realism is costly in error, and much of what is generated there is fabricated.

Suppose we allow at most $\tau$ units of extra squared error per pixel for the sake of realism. Then the realism level should be

$$\alpha(x) = \min\!\big(1,\ \sqrt{\tau / m(x)}\big).$$

This gives full sharpness wherever uncertainty is below the budget, and realism that fades as $1/\sqrt{m}$ in darker, noisier regions.

The claim is that training a low-light codec end to end under this spatially varying objective produces a better operating point than any global setting can reach. The bits must be trained jointly with the rule, because the value of a bit now depends on how much realism it unlocks. In a mid-SNR region, a bit that lowers $m$ also raises the permitted $\alpha$, so it buys both fidelity and sharpness.

Relation to Prior Work

Three papers motivate the proposal by exposing the gap.

Jointly Optimizing Image Compression with Low-light Image Enhancement is the only joint low-light codec in the file. It has an SNR-aware branch that guides features with a spatial signal-to-noise map. It trains purely for rate versus PSNR and MS-SSIM, so at its operating points near 0.06 bits per pixel it is pushed toward blurry averages. We keep its joint design and add the missing perception axis.

Decouple Distortion from Perception: Region Adaptive Diffusion for Extreme-low Bitrate Perception Image Compression does make the tradeoff spatial. It transmits only regions of interest and lets a Stable Diffusion decoder fill the rest. Its split is binary, its regions come from edges or boxes, and it is tested only on well-exposed images. In near-black scenes, shot noise produces false edges everywhere, so its masks would point at noise. We replace the binary mask with a continuous, uncertainty-derived control.

Training-Free Rate-Distortion-Perception Traversal With Diffusion and Conditional Rate-Distortion-Perception Trade-Off study how to move along the tradeoff, but treat the operating point as one global knob.

Two lines of low-light work supply the method and also constrain it.

SNR-Aware Low-light Image Enhancement shows that SNR varies sharply within a single frame. It uses that variation to route computation between convolution and attention, but not to set bits or realism. Its SNR map is our cheapest proxy for $m(x)$.

Diff-Retinex and Low-Light Image Enhancement via Generative Perceptual Priors show that diffusion decoders improve FID and LPIPS on low-light data while losing PSNR. That is the global tradeoff our method should split by region. They operate without compression, so they serve as quality ceilings rather than direct competitors.

For the bit-allocation mechanism we reuse the spatially varying quantization step sizes and two-stage soft-then-hard training of Soft then Hard.

The constraints come from two directions. Thinking in Granularity allocates precision by patch entropy. That works on clean images, but its entropy cue would reward shot noise, so we do not allocate by any input statistic. Bits are learned, and the only hand-derived quantity is the realism rule.

Finally, the metric-misalignment literature shapes how we evaluate. Gap-closing Matters built SQUARE-LOL, a dataset of human pairwise preferences on enhanced low-light images, and the IACA blind quality metric trained on it. Its results show that generic quality metrics misjudge low-light outputs. Work on texture-biased feature metrics (Xiaoshi Wu) shows FID can reward fabricated structure. For that reason, no learned perceptual metric is used as a training loss here. All are held out for evaluation, and hallucination is measured directly against long-exposure ground truth.

Research Program

The codec takes a short-exposure raw or sRGB frame and transmits a quantized latent. Following the joint design of Cai et al., it decodes directly to the enhanced, normally exposed image.

The decoder has two heads that share a backbone. The mean head is trained with squared error and estimates $\mu$. A light diffusion head is conditioned on the decoded latent and produces posterior samples $s$. A diffusion model is a generator that learns to turn random noise into images by repeated denoising.

A small uncertainty head predicts $m(x)$ from the decoded latent alone. It is trained to regress the squared residual of the mean head against ground truth. Because it reads only the latent, the decoder knows $\alpha(x)$ with no side information.

Training has two stages. The first stage trains rate plus squared error, so that $\mu$ and $m$ become meaningful. The second stage adds the blended objective: the rate term, the distortion of the $\alpha$-blended output, and a patch-wise realism term from a discriminator weighted by $\alpha(x)$. Gradients flow into the encoder's per-element quantization scales, so bit allocation adapts to the spatial objective. The budget $\tau$ is the single user-facing control, replacing the global perception weight.

We will also study what the codec learns. We will map where bits land as a function of local illumination and noise, compared with a globally trained codec of the same architecture. We will check whether the learned allocation concentrates bits in mid-SNR regions, as the joint argument above predicts. A cheap variant replaces the learned $m$ with the SNR map of Xu et al. This tests whether physics-based noise estimates suffice.

Baselines

The naive reference is the two-step pipeline. One version compresses the dark image with a standard codec and then enhances it. The other enhances first, then compresses. Cai et al. argue that both are worse than joint coding. The first amplifies compression artifacts and the second wastes bits on amplified noise. They also report the joint model needs about 494.56G FLOPs versus over 1100G sequentially. We run both versions with SNR-Aware as the enhancer and the Cheng-style learned codec, at matched bitrates.

The strongest existing method on this exact task is the Cai et al. joint codec, retrained under our protocol. The file gives its qualitative advantage near 0.06 bits per pixel on SID but no PSNR at a fixed rate. No reported result was found on Lacuna for its PSNR–bitrate numbers, so we will reproduce its curve ourselves.

The decisive competitor is stronger than Cai et al. It is the same architecture as ours, given the same diffusion decoder and realism loss, but trained with a single global perception weight. We sweep that weight across at least six values with the same compute and tuning budget as our $\tau$ sweep. This family traces the global tradeoff surface.

Beating it means one of two things. Either a single $\tau$ setting achieves lower shadow-region error and better lit-region realism than every member of the global family at equal bitrate, or our method's points lie outside the convex hull of the family's points in the space of bitrate, shadow PSNR and lit-region LPIPS.

We also run MRIDC's binary region-of-interest scheme on our backbone, with its masks computed from enhanced edges. Two ablations isolate the key ingredient. The first applies the $\alpha$ rule only at decode time to the globally trained codec, with no joint bit training. The second replaces the learned $m$ with the SNR map.

Diff-Retinex and GPP-LLIE run uncompressed as ceilings. Diff-Retinex reports FID 47.85 and LPIPS 0.048 on LOL. SNR-Aware reports 24.61 dB PSNR on LOL-v1, also uncompressed.

Why not just…?

Why not just add a perceptual or adversarial loss to the Cai et al. codec and tune its weight? This is the global-weight family above, and it is the real threat. The local error formula predicts that any single weight either under-sharpens lit regions or over-invents in shadows. That prediction can fail. If the low-light images in SID and LOL have uncertainty that is fairly uniform after enhancement, a global weight will do as well. The hull-dominance test settles it. If no point escapes the global family's hull, we will report that spatial control buys nothing on these benchmarks.

Why not just use a distortion-only codec and run a spatially weighted diffusion enhancer afterward? This is the decode-time-only ablation, and it could plausibly match us. It is the reason we claim the realism rule itself as the core idea and joint bit training as secondary. If decode-time blending closes most of the gap, the practical recommendation becomes "keep your codec, change your decoder." That is still a useful and cheaper result, and we will state it plainly rather than insist on joint training.

Why not just use the SNR map as a binary region-of-interest mask, as MRIDC does with edges? A binary mask has no gradient between "trust" and "invent." Medium-SNR regions are where the file's sources locate most low-light difficulty, and a binary mask must misclassify them one way or the other. The MRIDC-style baseline and the SNR-map ablation measure this directly. If the continuous map beats the binary one only marginally, the simpler mask is the better engineering choice.

Why not just let end-to-end training learn bit allocation without any hand rule? Our bits are learned. The only fixed component is how the realism weight depends on predicted uncertainty. The global-weight family is exactly "learn everything with a global objective." What we add is a spatial shape for the objective, derived from the local error identity, which a global scalar cannot express.

Proposed Experiments

We train on SID Sony raw data at the ×100, ×250 and ×300 exposure ratios (Learning to See in the Dark) and on LOL-v1 and LOL-v2-real. SDSD and SMID are held out to test transfer. Bitrates span roughly 0.02 to 0.3 bits per pixel, so they cover the 0.06 regime of Cai et al.

Every image is divided into SNR bins using the long-exposure reference. We report PSNR and MS-SSIM within each bin and LPIPS within each bin, alongside whole-image FID, NIQE and IACA.

Hallucination is measured directly. On SID we compute false-edge rate, meaning edges present in the output but absent in the long-exposure reference, restricted to the lowest-SNR bin. We also use the fact that several short exposures exist per scene. Content that varies across decodes of different short exposures, yet is absent from the reference, is invented by construction.

Human judgment comes from a pairwise study following the SQUARE-LOL protocol. Raters compare our codec against the best global-weight codec at matched rates. We also measure how well IACA, LPIPS and FID agree with those judgments on compressed outputs. IACA reports a rank correlation of 0.875 on SQUARE-LOL's enhanced, uncompressed images. Whether it holds up under compression artifacts and fabricated texture is itself a finding about the gap between metrics and human preference.

The project succeeds if three things hold at equal bitrate. Our codec escapes the global family's hull. Its lowest-SNR false-edge rate falls below that of the global codec with the same lit-region LPIPS. Human raters prefer it. If only the decode-time ablation succeeds, we report the decoder rule as the contribution.