Sparse Atoms of the Perception-Distortion Gap

When an image restorer sharpens a blurry photo, every detail it adds beyond the safe, blurry average costs accuracy. Whether that cost is worth paying depends on which detail it is. A plausible fur texture makes the image look real for a small accuracy penalty. An invented letter on a sign costs about as much accuracy and buys little realism. Current tools cannot tell these apart. They blend a blurry accurate output with a sharp generated output using one global dial, so every invented detail is kept or removed in the same proportion. We propose to break the difference between the two outputs into a sparse dictionary of reusable visual pieces ("atoms") and learn how much of each atom to keep. The aim is to trace a better tradeoff curve than global blending and to show, atom by atom, which invented details are cheap and which are expensive.

The opportunity rests on a simple accounting fact. The extra error of any restorer, beyond the best achievable average-error restorer, equals the energy of its deviation from that average. Accuracy therefore works like a budget, and the open question is how to spend it. Uniform spending is provably optimal for one particular measure of realism. We suspect it is not optimal for the measures that track what people see. A sparse decomposition gives a natural set of items to allocate the budget over, and a natural vocabulary for auditing what gets hallucinated.

Core Claim

Some background terms first. The perception-distortion tradeoff is the result that a restorer cannot be both maximally accurate (low mean squared error to the true image) and maximally realistic (its outputs distributed like real images). The Perception-Distortion Tradeoff proved this tension holds for any distortion measure. A Theory of the Distortion-Perception Tradeoff in Wasserstein Space sharpened it for one choice of realism measure, the Wasserstein-2 distance between output and real-image distributions in pixel space. Under that measure, linearly blending the minimum-mean-squared-error (MMSE) output with a perfectly realistic output traces the optimal frontier exactly. A sparse autoencoder (SAE) is a model that rewrites a signal as a sum of a few vectors drawn from a large learned dictionary. It is used in interpretability to split tangled representations into single-concept directions.

The identity we build on is the following. Let $y$ be the degraded input, $x$ the true image, $\hat{x}_{\text{MMSE}} = \mathbb{E}[x \mid y]$ the MMSE output with error $D^*$, and $\hat{x}$ any restorer whose output depends only on $y$ (possibly with internal randomness). Because the MMSE error is orthogonal to every function of $y$,

$$\mathbb{E}\|x - \hat{x}\|^2 = D^* + \mathbb{E}\|\hat{x} - \hat{x}_{\text{MMSE}}\|^2 .$$

Every point on any tradeoff curve is thus a choice of a "residual" $r(y) = \hat{x} - \hat{x}_{\text{MMSE}}$, and its distortion price is exactly the residual's energy. We decompose the residual between an MMSE-type restorer and a perceptual restorer into sparse atoms, $r \approx \sum_k f_k(y)\, d_k$. Here $d_k$ are dictionary images placed convolutionally and $f_k$ are sparse activations. We then learn a gain $g_k \in [0,1]$ per atom:

$$\min_{g}\; \mathrm{Div}\big(p_{\hat{x}_g},\, p_x\big) \quad \text{s.t.}\quad \mathbb{E}\Big\|\sum_k g_k f_k(y)\, d_k\Big\|^2 \le B, \qquad \hat{x}_g = \hat{x}_{\text{MMSE}} + \sum_k g_k f_k(y)\, d_k .$$

In this objective, $\mathrm{Div}$ is a realism divergence and $B$ is the distortion budget. Sweeping $B$ traces a curve.

The claim has two parts. First, when $\mathrm{Div}$ is pixel-space Wasserstein-2, the optimal gains should come out nearly uniform, recovering Freirich et al.'s result. This is a built-in check that the machinery is correct. Second, when $\mathrm{Div}$ is a divergence over deep-feature or local-patch statistics, which is closer to what FID-style metrics and human judges measure, the optimal gains should be strongly non-uniform. The resulting curve should then dominate uniform blending. The learned gains also act as a price list, a ranked inventory of which atoms buy realism cheaply and which are costly hallucinations.

Relation to Prior Work

Two restoration papers motivate the need for finer control than a global scalar, and both supply baselines. CTSR distills a fidelity teacher (ResShift) and a realness teacher (OSEDiff) into a one-step student whose timestep acts as a continuous slider. Flexible Style Image Super-Resolution using Conditional Objective (FxSR) trains one network conditioned on a style value, or a spatial map of style values, that reweights pixel, perceptual and adversarial losses. Both move along the tradeoff with controls that act on whole loss terms or whole trajectories, either globally (CTSR) or per pixel region (FxSR). Neither decomposes what is being added. Our proposal acts on the content of the residual itself, and it needs no retraining of the restorers.

The Wasserstein theory of Freirich et al. constrains the proposal and is not an obstacle to it. It says that atom selection cannot beat uniform blending when realism is pixel-space Wasserstein-2. We therefore make no such claim; we test it as a sanity check. Any gain we report must come from a realism measure that differs from pixel-space Wasserstein-2. The file notes that this measure does not track human-preference metrics well, which is exactly the gap we exploit. FxSR also reports halo and ghosting artifacts from blending sharp and smooth edges. That is a second reason to expect uniform blending to underperform on perceptual measures.

The sparse decomposition side supplies the method. Sparse Relaxed-Lasso Steering selects a small support of SAE features in text-to-image diffusion models and refits their strengths. It shows that sparse selection (about 300 features) beats dense steering (about 7,800 features) on an alignment, consistency and quality score (3.97 versus 3.89 for SAEdit and 3.32 for all features). We borrow its select-then-refit logic. We differ in three ways: the task is restoration and not editing, the decomposition is over the restoration residual with a linear pixel decoder so that distortion cost is exactly computable, and the objective is a formal tradeoff curve. Sparse Feature Circuits suppresses unwanted behavior by ablating a few identified features (SHIFT). Our gain-to-zero atoms are the restoration analogue, chosen by an explicit budget and not by human inspection.

PatchSAE shows that patch-level SAEs on vision models give monosemantic visual concepts, so patch-level atoms are a reasonable unit. Causal Interpretation of Sparse Autoencoder Features in Vision warns of a failure mode: features in attention models often fire away from the content they describe (about 14% of layer-22 features in CLIP-ViT-L/14). This is why we keep the decoder convolutional and pixel-local. Each atom's pixel footprint is then fixed by construction and cannot leak into backgrounds through attention. A Theory of Atomic Features reports that SAE features largely persist across dictionary sizes, 79.0% from 4,096 to 65,536 at threshold 0.7. That result motivates a check that our price list is stable across dictionary sizes, so that it reflects real structure and not an artifact of one training run.

Research Program

We first construct restorer pairs. On synthetic 4× super-resolution, an MSE-trained EDSR serves as the MMSE proxy and ESRGAN serves as the perceptual restorer, the same pair Freirich et al. analyze on BSD100. On real-world super-resolution, ResShift and OSEDiff serve as the pair, which are CTSR's own teachers. This lets us compare against CTSR on the restorers it was built from. We train on DIV2K.

We then train the residual dictionary. It is a convolutional SAE whose decoder is linear in pixels, with dictionary images of size around 16×16 at two or three scales and a Top-K sparsity constraint. Its encoder sees the residual patch together with context features from the MMSE output. The context lets atoms tie to content ("fur", "brick", "stroke-like text") and not only to frequency. Whatever the SAE fails to explain is carried by a separate uniform gain, so no part of the residual escapes the accounting.

Next we learn the gains under the budget, using a Lagrangian sweep and a differentiable realism divergence. The divergence is a sliced Wasserstein or kernel distance between deep features of output patches and real high-resolution patches. Optimizing against one feature extractor invites metric gaming. We therefore evaluate with a different extractor (FID), with the no-reference NIQE score, with LPIPS reported as a reference-based perceptual distance, and with a small paired human preference study at matched PSNR.

Finally, we analyze the price list. We rank atoms by realism gained per unit of energy and visualize the top and bottom atoms. We test whether expensive atoms coincide with content that varies across posterior samples from OSEDiff, measured by activation variance over repeated draws. We also test whether the ranking is stable across dictionary sizes and data splits, following the stability tests of the atomic-features theory.

Baselines

The naive reference points are the two endpoints themselves: the MMSE-proxy output (EDSR or ResShift) and the perceptual output (ESRGAN or OSEDiff). Every method must sit between or beyond these, and they anchor the PSNR range of the curves.

The strongest training-free competitor is uniform pixel blending, the provably optimal method of Freirich et al. for pixel-space Wasserstein-2. We sweep the blend weight on the identical restorer pair so that it uses the same two outputs as our method. Beating it means the gain-allocated curve lies below the blending curve on FID and patch-feature divergence across a range of PSNR budgets, measured on held-out BSD100, General100 and RealSR. The equations make a prediction here: on pixel-space Wasserstein-2 we expect to tie, and a tie there is a pass, not a failure. The file has no reported numbers for blending on BSD100 ("no reported result found on Lacuna for" EDSR/ESRGAN blending metrics), so we run it ourselves.

The strongest trained method is CTSR. Its reported result on RealSR at $t=0.2$ is a PSNR of 26.29 with an LPIPS of 0.3210. We run its slider across $t$ on RealSR. Beating it means better FID, NIQE and LPIPS than CTSR at matched PSNR, and specifically LPIPS below 0.3210 at a PSNR of at least 26.29. We start from its own teachers and spend no extra training on the restorers, only on the dictionary and gains, so any advantage cannot come from stronger base models. FxSR is the second trained baseline, run with both its global and its spatial style map on DIV2K-trained models evaluated on BSD100 and General100. The file reports only qualitative claims for FxSR, with no reported number on Lacuna for its PSNR or LPIPS, so we retrain or rerun it under the same data.

Three ablations isolate the key ingredient, which is allocation over sparse content-aware atoms. The first is spatial blending, a per-pixel blend weight optimized with the same objective and budget. The second is wavelet-subband gains, one gain per subband and orientation. The third is dense principal-component atoms of the residual, the same number of gains with no sparsity, mirroring the dense steering baselines in Sparse Relaxed-Lasso Steering. All three share the optimizer, the realism divergence, the budget sweep and the parameter count, matched as closely as possible. The proposal succeeds only if sparse atoms beat all three, not just uniform blending.

Why not just…?

Why not just blend the two outputs in pixel space, since that is provably optimal? It is optimal only for pixel-space Wasserstein-2 realism. We include that case as a check and expect a tie there. FxSR documents halo and ghosting artifacts from blending misaligned sharp and smooth edges, which pixel Wasserstein-2 does not penalize but feature-based realism measures and human viewers do. If uniform blending also matches us on feature-space divergence and on human preference at equal PSNR, the central claim is false, and the experiment is designed to show it.

Why not just use a spatial blend map, as FxSR does with its style map? This is a serious threat. Much of the benefit may simply be "sharpen textures, keep flat regions smooth," and a per-pixel weight can do that. A spatial map cannot, however, keep a plausible texture while dropping an invented stroke that sits on the same pixels. Atoms can. The spatial-blending ablation uses the same objective and budget. If it matches sparse atoms, we will report that the gain is spatial and not content-based, and drop the content claim.

Why not just allocate the budget over frequency bands? The common intuition is that hallucination equals high frequency. The wavelet-subband ablation tests this directly. If subbands match atoms, the SAE adds interpretability but no performance, and we would say so. Our expectation is that hallucinated and plausible details share frequencies but differ in structure, which a subband gain cannot separate.

Why not just steer with dense directions such as principal components or mean differences? In text-to-image editing, dense selection degraded the combined alignment, consistency and quality score from 3.97 to 3.32 because edits leaked across unrelated content (Sparse Relaxed-Lasso Steering). The dense principal-component ablation tests whether the same holds when the target is a distortion budget and not an edit.

Why not just use CTSR's slider? It is fast and strong. But it is one global knob over a learned trajectory, so it cannot remove a specific invented detail without also softening everything else. Its reported RealSR operating point is the target to beat at matched PSNR.

Decisive Checks

The first check is the theory check. With pixel-space Wasserstein-2 as the divergence, run on downsampled images where the distance can be estimated reliably, the learned gains must come out near-uniform, and the curve must coincide with uniform blending. If they do not, the optimizer or the accounting is wrong.

The main result is a dominance test. The sparse-atom curve must lie below uniform blending, spatial blending, wavelet gains and dense principal components on held-out FID and patch-feature divergence at matched PSNR. On RealSR it must reach LPIPS below 0.3210 at a PSNR of at least 26.29 against CTSR, and human raters must prefer it at equal PSNR.

The interpretability result is that the price list is stable across dictionary sizes and data splits. A further test is that expensive atoms have high activation variance across OSEDiff posterior samples, meaning they are the details the input does not pin down. If the dominance test fails while the price list is stable, the work still delivers an audit tool for which content a restorer hallucinates and what that content costs. That would be a smaller but honest contribution.