A lossy compressor can serve as a denoiser that never sees a clean image. Natural images are predictable, so they are cheap to encode. Random noise is not, so a codec on a tight bit budget keeps the image and drops the grain. Zero-shot neural compression denoisers fit a small learned codec to a single noisy photo and are now competitive with trained methods. Because they minimize squared error, though, they output the average of all plausible clean images, which is smooth, waxy and visibly blurrier than a real photograph. The obvious fix is to add a "realism" term. That fix runs into the central problem of perceptual quality measurement: realism is normally judged either against a clean reference that does not exist here, or by learned no-reference scorers that reward any sharp texture, including invented texture.
This note proposes that, for denoising with a known noise model, realism can be checked without any reference or human labels. Suppose an output is a true sample from the distribution of clean images consistent with the noisy one, called the posterior. Then what it removes from the input, the residual, must look exactly like the noise and must be statistically independent of the output. The converse also holds. Blurry averages fail this test, because lost edges leak into the residual. Hallucinating generators fail it too, because the invented texture shows up, sign-flipped, in the residual. We propose to use this residual-independence criterion in three ways: as a training signal for a stochastic decoder on top of a zero-shot compression denoiser, as a ground-truth-free way to set the codec's bit budget, and as a reference-free quality score whose agreement with human judgment we test directly.
Core Claim
Let $Y = X + N$, where $X$ is the clean image and $N$ is noise with a known law, independent of $X$, whose characteristic function never vanishes (true for Gaussian noise). Let $\tilde X$ be any randomized estimate computed from $Y$. Then
$$\tilde X \sim p(X \mid Y) \quad\Longleftrightarrow\quad Y - \tilde X \stackrel{d}{=} N \ \text{ and } \ (Y - \tilde X) \perp \tilde X .$$
The proof is short. If the residual is noise-distributed and independent of $\tilde X$, then $Y = \tilde X + E$ with independent $E$. Deconvolution identifiability then forces $\tilde X$ to share the marginal law of $X$, so the pair $(\tilde X, Y)$ has the same joint law as $(X, Y)$. The forward direction follows from the same equality in law.
The right-hand side never mentions the clean distribution. "Perfect perceptual quality", which The Perception-Distortion Tradeoff defines as the output distribution matching the natural-image distribution, therefore becomes testable from noisy data alone in the denoising setting. The criterion also makes a quantitative prediction. A posterior sample has exactly twice the squared error of the posterior mean, which is the same "MSE doubling" that Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff bounds and Optimally Controllable Perceptual Lossy Compression observes on MNIST. That ratio can itself be measured without ground truth using the unbiased noisy-data error estimator of Evaluating Unsupervised Denoising Requires Unsupervised Metrics.
The equivalence holds at the level of distributions, and in practice we see one image. We therefore estimate both conditions over co-located patches across spatial positions, which assumes local stationarity. Testing whether that finite-sample, single-image estimate still tracks perceptual quality is the empirical core of the project.
Research Program
The method side builds on Zero-shot Denoising via Neural Compression: Theoretical and algorithmic framework (ZS-NCD), which fits an untrained neural codec to overlapping patches of one noisy image under a squared-error-plus-rate objective, and on DeCompress: Denoising via Neural Compression, which shows the same single-image effect and treats the rate weight $\lambda$ as a denoising knob tied to the noise level. Both converge to the blurry posterior-mean regime. We keep their encoder and entropy bottleneck unchanged.
On top of that bottleneck we add a second, stochastic decoder $G_s(z, u)$. It takes the compressed code $z$ and fresh random input $u$ and is trained only with
$$\mathcal{L}_s = \mathrm{MMD}\big(\text{patches of } Y - \tilde X,\ \text{patches of } N\big) + \mu \,\mathrm{HSIC}\big(Y - \tilde X,\ \tilde X\big), \qquad \tilde X = G_s(z,u).$$
Here MMD is a kernel two-sample distance between residual patches and freshly simulated noise patches. HSIC is a kernel dependence measure between co-located residual and output patches. The weight $\mu$ balances the two terms.
Sharing one encoder across the mean decoder and the sample decoder follows the result of Optimally Controllable Perceptual Lossy Compression that a squared-error-trained encoder is optimal for every point on the distortion-perception curve. Linear interpolation between the two decoders then traces that curve. That result assumes clean inputs, so whether it survives noisy inputs is something we test, not assume.
The randomness design draws on Optimal Neural Compressors for the Rate-Distortion-Perception Tradeoff. That work shows deterministic decoders cannot reach realism at low rate and that shared dither helps. We compare private random input against dithered latents.
The criterion also addresses the unsolved problem of choosing $\lambda$ without a reference. If the rate is too high, noise passes through the code into $\tilde X$, so the residual is weaker than the noise and correlated with the output. If the rate is too low, real structure is discarded and appears in the residual. The residual statistic should therefore be U-shaped in $\lambda$, and its minimum should sit near the oracle-best $\lambda$.
Separately, we treat the residual statistic, evaluated on any denoiser's output, as a reference-free quality score. We compare it with full-reference perceptual metrics, which need the clean image, and with no-reference scorers, which do not.
Baselines
The naive reference points are the noisy input and the ZS-NCD posterior-mean output itself. ZS-NCD reports 30.60 dB PSNR on Kodak24 at Gaussian $\sigma=25$, against 28.90 dB for Deep Image Prior and 29.54 dB for Zero-Shot Noise2Noise. Our sample decoder is expected to lose PSNR, roughly 3 dB if it truly samples the posterior. It "beats" this baseline only if humans prefer it and its measured error ratio to the mean sits near two, not far above.
The strongest perceptual competitor is Test-Time Preference Optimization for Image Restoration (TTPO). It refines a squared-error restoration with diffusion-generated candidates ranked by no-reference quality scorers. It reports a MUSIQ score rising from 73.79 to 74.24 for SwinIR on Kodak24 denoising, and 70.31% expert preference over the base model. We apply TTPO on top of the same ZS-NCD output, so the starting point is identical, and we report wall-clock time alongside it.
Our win condition against TTPO is not a higher MUSIQ score. MUSIQ is one of the scorers TTPO optimizes, so MUSIQ is compromised as a judge in this comparison. Instead, we must match or exceed TTPO in a blind human study on natural images while showing lower hallucination, measured by full-reference error and residual dependence.
We also include Training-Free Rate-Distortion-Perception Traversal With Diffusion. It traverses the perception axis with a score-scaled diffusion sampler and is the closest "posterior sampler with a pretrained prior". No reported denoising result was found on Lacuna for it, so we run it ourselves with matched noise levels.
Optimal Transport for Unsupervised Restoration Learning needs an unpaired clean corpus. No reported numerical result was found on Lacuna for it, and we include it only where such a corpus exists. A supervised SwinIR serves as a fidelity ceiling.
Ablations isolate each ingredient. Removing the residual loss recovers ZS-NCD. Removing the bottleneck tests whether the residual loss alone suffices, which we expect it does not, because single-image finite-sample tests can be fooled. Dropping either the MMD term or the HSIC term isolates each half of the characterization. All zero-shot methods receive the same per-image GPU budget and the same $\lambda$ search grid.
Why not just…?
Why not just add a fraction of the noisy input back to the smooth output? This "grain retention" trick is cheap, and humans often like it. It is deterministic, however, and its residual is a scaled copy of the mean-estimator's residual, so it keeps the edge structure that blurring left behind. The criterion predicts a failed independence test. We run it at the blend weight that best matches the noise variance and let the human study decide whether the extra machinery is worth it.
Why not add independent noise of the right variance to the posterior mean? This is the most serious threat. For Gaussian signals it is exactly a posterior sample. If natural-image posteriors were nearly Gaussian, it would match our method at a fraction of the cost. Our bet is that real posteriors are strongly non-Gaussian along edges and textures. To settle this, we run both a white-noise version and a version shaped by per-pixel uncertainty, estimated from the spread of independent ZS-NCD runs. If either matches us in human preference and in the error-ratio check, we would report that the residual criterion is a metric contribution rather than a method contribution.
Why not use a pretrained generative refiner such as TTPO or a large multimodal restorer? These carry a strong clean-image prior and may win on natural photographs. Is Nano Banana Pro a Low-Level Vision All-Rounder? shows that such generators trail specialist denoisers by 5 to 8 dB PSNR because they replace noise with invented content. This is unacceptable in microscopy, where no matching prior exists. Our microscopy experiments and the hallucination checks below test this directly. We concede natural photographs if the human study goes against us there.
Why not judge realism with existing no-reference or full-reference metrics? No-reference scorers reward sharp texture whether or not it is real. Full-reference metrics such as BiRQA, PAMSE and PIM need the clean image, which zero-shot settings lack. The noisy-data error estimates uMSE and uPSNR, from the unsupervised-metrics paper, inherit squared error's preference for blur. We use the full-reference metrics only where clean images exist, to validate our reference-free score.
Proposed Experiments
The first decisive check validates the metric. On Kodak24 and BSD68 with Gaussian noise at $\sigma = 15, 25, 50$, we collect outputs from roughly ten denoisers spanning blurry, grainy and hallucinating behaviour. We run a two-alternative forced-choice human study and correlate rankings by Spearman correlation with our residual score, with MUSIQ and NIQE, and with BiRQA, PIM, PAMSE and LPIPS computed against the clean image. For context, BiRQA reaches 0.988 Spearman correlation on LIVE and PAMSE 0.9162 on TID2008, though those benchmarks have no noisy-input pairing.
Success means our score, using no reference, tracks human preference about as well as the full-reference metrics and clearly better than the no-reference scorers on the hallucinating subset. We also deliberately inject plausible fake textures into outputs. The residual score should flag them while MUSIQ rewards them.
The second check tests the theory. The ratio of the sample decoder's measured error to the mean decoder's should approach two. Averaging many samples should converge to the mean output. Both are measured with uMSE on held-out noisy realizations, so they also apply to real data.
The third check tests $\lambda$ selection. We look for a U-shaped residual score whose minimum lands within the oracle-PSNR range, for every noise level and for the Poisson case.
Finally, we move to calibrated Poisson-Gaussian noise on the Mouse Nuclei and FMD microscopy data and on PolyU, comparing against TTPO and the diffusion sampler. Spatially correlated or fixed-pattern noise breaks the known-noise assumption, as it breaks uMSE, and we report those failures. We also report runtime honestly. ZS-NCD already takes 40 to 50 minutes per high-resolution image, and the second decoder adds to that.
References
- The Perception-Distortion Tradeoff
- Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff
- Optimally Controllable Perceptual Lossy Compression
- Evaluating Unsupervised Denoising Requires Unsupervised Metrics
- Zero-shot Denoising via Neural Compression: Theoretical and algorithmic framework
- DeCompress: Denoising via Neural Compression
- Optimal Neural Compressors for the Rate-Distortion-Perception Tradeoff
- Zero-Shot Noise2Noise: Efficient Image Denoising without any Data
- Test-Time Preference Optimization for Image Restoration
- Training-Free Rate-Distortion-Perception Traversal With Diffusion
- Optimal Transport for Unsupervised Restoration Learning
- Is Nano Banana Pro a Low-Level Vision All-Rounder? A Comprehensive Evaluation on 14 Tasks and 40 Datasets
- BiRQA: Bidirectional Robust Quality Assessment for Images
- Perceptual Fidelity Aware Mean Squared Error
- An Unsupervised Information-Theoretic Perceptual Quality Metric