Every super-resolution model has to decide how much detail to invent. If it predicts the average of all high-resolution images consistent with the blurry input, it stays close to the truth in pixel error but looks smooth. If it samples one sharp, plausible answer, it looks real but makes things up. Today that decision is made once for the whole image, through a single slider or loss weight. We propose making it separately for each internal feature. A sparse autoencoder is a network that rewrites a dense activation vector as a small number of active entries drawn from a large dictionary of directions, each ideally meaning one thing. Once a restoration model's activations are rewritten this way, we can measure, feature by feature, how strongly the low-resolution input pins that feature down. We then let the generative model supply only the features the input leaves undetermined, and copy the input-determined features from a fidelity-oriented estimate.
The central claim is testable. When "what the input determines" and "what the prior invents" are separated in a sparse, roughly independent feature basis, gating realism per feature should reach a better perception–distortion frontier than three alternatives: a global slider, a spatial mask over image regions, or the same gating in a dense PCA basis. The gains should concentrate where structure and texture share the same pixels, such as text printed on fabric, hair over facial contours, or patterned architecture. Spatial and frequency masks cannot separate those cases, because they operate on regions or frequency bands, not on what a feature means.
Background and what is actually missing
The Perception-Distortion Tradeoff proves that forcing a restorer's output distribution toward natural images raises the minimum achievable distortion. A Theory of the Distortion-Perception Tradeoff in Wasserstein Space recasts that bound in terms of Wasserstein distances between distributions. Both results treat the tradeoff as a single global curve in image space. That framing motivates this proposal but does not constrain how an estimator distributes its "realism budget" across image components.
Existing controllable methods all spend the budget globally. CTSR distills a fidelity teacher (ResShift) and a realness teacher (OSEDiff) into a one-step student whose timestep $t$ acts as one slider for the whole image. MOBOSR searches global loss weights with multi-objective Bayesian optimization and yields a static model per trade-off point. DiffBIR comes closest to local control: it modulates guidance by spatial frequency. But text strokes and grass blades are both high-frequency, so a frequency mask cannot tell them apart.
SemanticSRJudge supplies the empirical motivation. It shows content-specific reversals, where diffusion models like SUPIR win on faces and fur but lose to regression or GAN models on text and patterns. It diagnoses the problem but offers no fix.
On the sparse-feature side, three papers supply the method. Sparse Feature Circuits shows that sparse-autoencoder features give far more compact causal handles than neurons (100 features match 1,500 neurons in Pythia-70M). Sparse Relaxed-Lasso Steering shows that principled sparse selection of such features in diffusion image models beats heuristic top-$k$ selection. Its setting is prompt-driven editing, not inverse problems, and it never measures distortion against a ground truth. A Theory of Atomic Features proves recovery of frequent atomic features under pairwise independence. That independence is exactly the property per-feature gating needs, and the same paper admits it only holds approximately in natural data.
The obvious intersection would be "train a sparse autoencoder on a diffusion restorer, find hallucination features by attribution, clamp them on text." That is Gap 3 in the file, and it has a weakness: it needs labels saying what counts as hallucination, and it reduces to a content-specific patch. Our proposal differs in two ways. The quantity we gate on is defined by the restoration problem itself, namely how much a feature varies across plausible restorations of the same input. And the output is a whole trade-off curve, not a single edit.
Core Claim
Let $f_i$ be the activation of sparse-autoencoder feature $i$ at some location, $y$ the low-resolution input, and $x$ the true image. We define a determinacy ratio as the share of a feature's natural variability that remains after seeing the input:
$$r_i = \frac{\mathbb{E}_y[\mathrm{Var}(f_i \mid y)]}{\mathrm{Var}(f_i)}.$$
Here the conditional variance is estimated across multiple stochastic diffusion restorations of the same $y$, and the denominator is the feature's variance over clean images. A value near 0 means the input fixes the feature, as with an edge position or a character's stroke topology. A value near 1 means the feature is supplied by the prior, as with the exact arrangement of fur strands.
For a single coordinate, replacing the posterior mean with a posterior sample doubles its expected squared error, from $\mathrm{Var}(f_i\mid y)$ to $2\,\mathrm{Var}(f_i\mid y)$. The distortion cost of "being realistic" in feature $i$ is therefore roughly proportional to $r_i\,\mathrm{Var}(f_i)$. Two conditions make features with low $r_i$ cheap to leave faithful and features with high $r_i$ the cheapest places to buy realism. The first is that features are close to independent, which is the atomic-feature assumption. The second is that decoder directions are not too correlated.
Our estimator blends a fidelity output (feature values $f^{\text{fid}}$) with a realness output (feature values $f^{\text{real}}$) as
$$\hat f_i = \alpha_i(\tau)\, f_i^{\text{real}} + (1-\alpha_i(\tau))\, f_i^{\text{fid}}, \qquad \alpha_i(\tau)=\sigma\!\big(k\,(r_i-\tau)\big).$$
In this blend, $\sigma$ is the logistic function, $k$ is a fixed sharpness, and $\tau$ is a single user-facing threshold. The user still turns one knob, but the knob moves features in order of how undetermined they are, not uniformly.
Research Program
We build on components the file already describes. The fidelity path is ResShift or an MSE-trained SwinIR, as in DiffBIR's first stage. The realness path is OSEDiff, which is one-step. Both produce images that can be pushed through the Stable Diffusion variational autoencoder, the image compressor used by these models. We train Top-$K$ sparse autoencoders on activations from a mid-level block of that autoencoder's decoder, which has hundreds of channels at 1/8 resolution. This gives a feature space shared by both paths and decodable back to pixels.
The sparse autoencoder's own reconstruction error is carried through unchanged as a residual. Blending then only moves the dictionary part, and the error term is shared by both paths, so it cannot masquerade as a gain. We also report the distortion this insertion adds with $\alpha\equiv0$ and $\alpha\equiv1$, which addresses the file's Gap 4 directly.
Determinacy ratios are estimated offline on DIV2K training crops. For each input we draw 8 to 16 seeds from a stochastic diffusion restorer (DiffBIR), encode each sample, and compute per-feature conditional variance. At inference, only the two one-step paths and the fixed $r_i$ table are used, so runtime stays near CTSR's regime rather than DiffBIR's 50 steps. We also test whether $r_i$ is stable across expansion factors, echoing the stability principle in the atomic-feature paper. Unstable ratios would signal that the gating rides on noise.
A second, analytic strand asks whether the features that the determinacy ratio ranks as prior-driven look like textures when visualized. Conversely, features the ratio ranks as input-determined should correspond to edges, strokes, and geometry. That check tells us whether the mechanism is semantic or only statistical. Either answer is informative.
Baselines
The naive reference points are the two endpoints and a plain pixel-space alpha blend of the fidelity and realness images, swept over the blending weight. Pixel blending of a mean-like estimate with a sample is the simplest estimator that traces a perception–distortion curve. Any method that cannot beat it has no claim.
The strongest existing method is CTSR swept over $t$. The number to beat comes from the CTSR paper: on RealSR at $t=0.2$ it reports PSNR 26.29 and LPIPS 0.3210, where LPIPS is a learned perceptual distance and lower is better. We will rerun the released CTSR model on identical RealSR, DIV2K-validation, and SemanticSR-Bench inputs rather than rely only on that reported pair. Beating it means that at CTSR's PSNR of 26.29 on RealSR, our curve reaches lower LPIPS, and that the hypervolume in the PSNR–LPIPS plane is larger over a common reference point, following the measure MOBOSR uses.
DiffBIR with region-adaptive guidance is the strongest local-control competitor. We run it at its default 50 steps and report its curve over guidance scale. Our method uses a different base model, so we additionally run DiffBIR's frequency mask inside our own two-path pipeline. That isolates the gating rule from the generator.
The key ablations hold everything fixed except the basis. The first replaces the sparse autoencoder with PCA on the same activations, with ratios computed per principal component. The second uses the raw channel basis. The third uses the sparse dictionary with $r_i$ randomly permuted, which keeps the sparse basis but breaks the determinacy ordering.
All methods share the same test images, the same metric code, and an equal tuning budget. That budget is one threshold sweep plus, for learned-selection variants, the same number of Bayesian-optimization evaluations.
Why not just…?
Why not just give CTSR a spatial map of $t$ from a text or segmentation detector? This is the most serious threat. If segmenting "text" and "texture" regions and lowering $t$ on text matches our results, the sparse autoencoder is unnecessary. We will build exactly this baseline with a per-pixel $t$ map. Masks come from SemanticSR-Bench's category structure plus an off-the-shelf segmenter, so the baseline is not starved of semantics. The claim survives only if our method wins specifically on patches where structure and texture overlap, for example text on textured backgrounds, patterns, and hair over faces. Our prediction is that it will. If the two tie there, we will report that region-level control suffices.
Why not just gate by per-pixel variance across diffusion samples? Pixel-space posterior variance also measures how undetermined each location is and needs no dictionary. Its weakness is that a location can hold a determined edge and an undetermined texture at once. Pixel variance then forces an all-or-nothing choice at that pixel. We implement this gate with the same samples and the same threshold sweep. If it matches the sparse version, the decomposition adds nothing, and the PCA ablation will already have hinted at this.
Why not just tune losses per trade-off point, as MOBOSR does? Loss reweighting yields a static model per point and cannot change behavior per feature at inference. It is also orthogonal to our method: a MOBOSR-trained fidelity model could serve as our fidelity path. No MOBOSR scalar is reported on RealSR in the file beyond a gain of over 5 dB in LR-PSNR against SROOE on DIV2K. We treat it as a possible input, not a competing curve.
Proposed Experiments
We evaluate on RealSR (Canon and Nikon splits), DIV2K-validation, Urban100 and Manga109 for structured content, and SemanticSR-Bench's seven categories. We report PSNR, SSIM, and LPIPS curves and PSNR–LPIPS hypervolume. On SemanticSR-Bench we add multi-layer DINOv2 semantic drift and the spatially grounded judge from SemanticSRJudge, so that hallucination is scored locally rather than by a global scalar. On the Text category we also report optical character recognition accuracy against the ground truth. The file reports no prior number for that measure, so it has no reference value.
Three checks decide the question. The first is whether, at matched PSNR, per-feature gating beats both the spatial-$t$ CTSR baseline and the pixel-variance gate on LPIPS and semantic drift in mixed-content patches. The second is whether that advantage disappears when the sparse basis is swapped for PCA or when $r_i$ is shuffled. That would show that sparsity and determinacy ordering, not just two-path blending, produce the gain. The third is whether the benefit grows on the categories where SemanticSRJudge found reversals, text and patterns, while faces and animals stay no worse than the realness endpoint. A result where only the first check passes would say region masks were the bottleneck. A result where all three pass would show that the perception–distortion tradeoff can be allocated feature by feature, and that sparse decomposition is what makes this allocation possible.
References
- The Perception-Distortion Tradeoff
- A Theory of the Distortion-Perception Tradeoff in Wasserstein Space
- CTSR: Controllable Fidelity-Realness Trade-off Distillation for Real-World Image Super Resolution
- Perceptual-Distortion Balanced Image Super-Resolution is a Multi-Objective Optimization Problem
- DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior
- SemanticSRJudge: Spatially-Grounded VLM Evaluation for Super-Resolution Quality Assessment
- Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models
- Sparse Relaxed-Lasso Steering: Automatic Sparse Autoencoder Feature Selection for Precise Image Editing
- A Theory of Atomic Features and Four Testable Predictions