Rain, snow and haze do not corrupt an image at one size. A single photograph can hold hairline streaks far from the camera, thumb-sized flakes close to it, and a veil of fog spread over everything. Restoration networks nonetheless process every image at the same hand-picked set of scales: fixed kernel sizes, fixed pooling factors, fixed pyramid levels. When an image contains artifacts outside that set, such as the unusually large, high-contrast snowflakes that unified models are reported to fail on, the network has nothing sized to handle them.
We propose that the network first work out, for each image, how many distinct artifact sizes are present and what those sizes are, and only then process the image at exactly those sizes. The counting step uses non-parametric Bayesian clustering, a family of methods that decide the number of clusters from the data rather than fixing it in advance. Its fast deterministic limit, known as small-variance asymptotics, reduces to "add a new cluster only if it pays for a fixed penalty." The processing step uses one convolutional operator whose weights are shared across all scales and whose scale is a continuous input. Adding a cluster therefore adds computation but no new parameters, and a scale never seen in training can still be instantiated.
Core Claim
The claim has three parts. The number and location of degradation scales can be inferred per image from a scale-space energy profile of the raw input residual. Allocating a shared-weight, continuously scaled operator to each inferred scale restores images better than fixed scale sets or fixed expert pools. The advantage should be largest when test artifacts are larger or more varied than those seen in training.
The last part is the falsifiable heart of the proposal. A network that only reweights a fixed menu of scales cannot reach sizes off the menu, whereas a network that reads the sizes off the input can. If that extrapolation advantage does not appear, the Bayesian machinery is not earning its place.
What Exists and How This Differs
Three lines of prior work motivate the gap by hard-coding scale. MSP-Former projects keys and values through preset multi-scale average pooling next to a local depth-wise branch. PRRN runs parallel kernels from 1×1 to 7×7 to align synthetic and real rain. AdaptIR inserts three fixed heterogeneous branches (convolutional, Fourier, and channel gating). All three choose the scales once for every input. We choose them per image and let their number vary.
Mixture-of-experts restorers route inputs to a fixed pool of specialist sub-networks. MoFME modulates one shared feed-forward block with lightweight experts and uncertainty-aware routing. M2Restore routes between Mamba and CNN experts using frozen DA-CLIP prompts. DirMoE makes routing differentiable with a Dirichlet selection-and-weighting scheme for language models. All three assume a finite expert count, and none indexes experts by spatial scale. We borrow one lesson from MoFME, that sharing one set of weights across experts avoids parameter bloat, and push it further: our experts differ only in a continuous scale value.
The closest Bayesian non-parametric work is Blind Image Denoising via Dependent Dirichlet Process Tree. It models clean 8×8 patches and noise as coupled Dirichlet-process mixtures solved by variational inference. It has no notion of scale hierarchy and no deep network. GRIDS is the closest deep work. It groups 11 degradation types into 4 groups by an offline binary search over KL divergences, and it motivates our proposal by showing that mixing conflicting degradations costs 2.24 dB on average. Its groups are fixed after training and are defined over degradation types, not over sizes within a single image.
The clustering method itself comes from DP-space. It derives a deterministic objective that penalizes the number of clusters $K$ and was shown fast on Hopkins 155 (about 2 seconds versus over 500 for sparse subspace clustering). That paper also constrains us. Its authors note that the cluster penalty governs granularity, so penalty sensitivity must be measured, not assumed away.
Research Program
Front end: a descriptor that keeps the degradation visible. PromptRestorer documents "degradation vanishing," the tendency of deep features to become clean and lose the corruption signature. It fixes this with a frozen VQGAN encoder. We instead compute scale statistics on a shallow signal: the input minus a cheap first-pass estimate of the clean image. For each local window we measure residual energy on a fine logarithmic grid of scales, using band-pass responses from a Laplacian-of-Gaussian stack. This yields a set of atoms indexed by $i$, each with a log-scale $s_i$ and an energy $e_i$.
Counting step. We minimize an energy-weighted version of the small-variance-limit objective:
$$ \min_{K,\ \mu_{1..K},\ z}\ \sum_i e_i\,(s_i - \mu_{z_i})^2 + \lambda K . $$
Here $\mu_k$ is the log-scale of cluster $k$, $z_i$ is the cluster assigned to atom $i$, and $\lambda$ is the price of opening a cluster. The objective is the limit of a Dirichlet-process mixture with vanishing variance, following DP-space, reduced to one dimension. It is solved with a few DP-means iterations per image in negligible time.
Weighting atoms by energy matters for the failure case in the file. A handful of large, high-contrast flakes carries enough residual energy to justify its own cluster even though such flakes are rare. We set $\lambda$ relative to the image's total residual energy, which makes it dimensionless, and we sweep it. As an ablation we also try a Pitman–Yor variant, whose prior favors many small clusters with a power-law tail. This follows the heavy-tail concern raised for Dirichlet processes in Bayes-QLAE.
Processing step. Each cluster $k$ instantiates one pass of a single shared-weight block. Features are resampled by factor $\exp(\mu_k)$, processed, and resampled back. Each pass is masked spatially by the soft responsibility of cluster $k$ at each location, and the passes are summed.
Gradients flow through the block and through the continuous scales $\mu_k$. The discrete assignments are recomputed per image and not backpropagated through, much as vector quantizers treat their nearest-code step. This keeps training end to end without Gibbs sampling. That matters because the file records 50 hours of Gibbs dictionary learning in the Beta-process super-resolution work.
Host network. We drop this module into MSP-Former in place of its fixed pooling branch. MSP-Former is small (2.83M parameters, 4.42 GMacs), so the scale module is a visible fraction of the network and its effect is easy to isolate.
Baselines
The naive reference is the unmodified host network with its preset pooling scales. This is MSP-Former as published, reported at 33.75 dB PSNR on CSD. We retrain it ourselves under our schedule so that every comparison shares data, iterations and augmentation.
The ablation that isolates the key ingredient keeps our shared-weight scaled operator but feeds it a fixed scale set. We test two versions: the geometric set {1, 2, 4, 8}, and a per-image K-means with $K$ fixed at the median $K$ our method infers. Neither version receives the counting step, so any gap measures the value of inferring how many scales and which ones.
The strongest structural competitor is a dense scale bank. The same operator is applied at 12 to 16 scales spanning the whole range, and softmax attention weights the outputs. We run it twice: once at matched FLOPs, which forces fewer scales, and once unconstrained. This is the alternative most likely to match us on in-distribution data.
The strongest routing competitor is an over-provisioned mixture of experts with DirMoE-style Dirichlet routing and its sparsity knob. Each of its experts is the shared operator with a learned, input-independent scale. Its learned sparsity could in principle discover an effective $K$ on its own.
Against published systems we report the strongest numbers the file contains. SmartAssign reports 29.45 dB PSNR on Snow100K-L (SmartAssign). PromptRestorer reports 37.48 dB on CSD. M2Restore reports 31.65 dB PSNR and 0.936 SSIM on the all-weather composite. PRRN reports a Brisque score of 24.57 on real DDN-SIRR rain. No reported result was found on Lacuna for SRRS PSNR or for Rain100H PSNR for these methods, so there we compare only against our retrained baselines.
We do not expect a 2.8M-parameter host to beat PromptRestorer's absolute CSD number. The success criterion is therefore relative. The method should beat the fixed-scale ablation and the matched-FLOP dense bank by a margin clearly above seed variance on Snow100K-L and the all-weather composite. It should beat all baselines by a larger margin on the scale-extrapolation test described below. As a secondary check, we transplant the module into a larger host and ask whether it closes part of the gap to SmartAssign on Snow100K-L.
Why not just…?
Why not just use a fixed mixture of experts with softmax routing? DirMoE reports that top-k softmax routing entangles selecting an expert with weighting it, yielding less specialized experts (41.13% versus 39.47% for Switch MoE on its zero-shot suite). That evidence comes from language, not images. The deeper problem is that each expert's scale is fixed at training time, so no routing decision can produce a size outside the pool. We include the strongest version of this alternative, over-provisioned Dirichlet routing, and the extrapolation test decides between it and our method.
Why not just cluster with K-means or a fixed-K Gaussian mixture? This is exactly our fixed-K ablation. DP-space shows that fixing $K$ turns model selection into a search problem. In our setting, images genuinely differ in how many artifact populations they contain: fog alone gives one, rain plus fog gives two. On synthetic mixtures with a known number of populations, we will measure whether inferred $K$ tracks the truth and whether fixed $K$ loses PSNR on images whose true count differs from it. If fixed $K$ loses nothing, the non-parametric part is unnecessary and we will say so.
Why not just rely on frozen CLIP or VQGAN embeddings? M2Restore and PromptRestorer do this, and PromptRestorer itself names dependence on the frozen encoder's training distribution as a limitation. These embeddings describe what a degradation is, not how large it is. Our descriptor is a calibrated measurement of size. We also test a variant that predicts scales from frozen DA-CLIP features, to show whether semantic embeddings can stand in for the scale-space measurement.
Why not just add more pyramid levels or dilated convolutions, or learn per-image scales with a regression head? This is the most serious threat. A dense bank with attention, or a small head that regresses $K_{max}$ scales plus gates, can approximate much of what clustering does in distribution. Our answer is not a priori. The dense bank must pay for every scale on every image, and a regressor trained on a bounded range of sizes has no reason to output sizes beyond it. The clustering step, by contrast, reads the size directly off measured energy. We run both alternatives at matched FLOPs. If either matches us on the extrapolation split, the per-image clustering claim fails.
Proposed Experiments
Decisive test: scale extrapolation. We synthesize rain streaks and snow masks at controlled size multipliers on clean images from the Snow100K and Rain100H training sources. Models train on multipliers between 0.5 and 1.5 and are tested at 2, 3 and 4. We also create mixtures with one to four populations of known size so we can score inferred $K$ and inferred $\mu_k$ against ground truth. We report PSNR and SSIM as a function of the multiplier. The prediction is that fixed-scale, dense-bank and fixed-expert baselines degrade sharply beyond 1.5 while our method degrades gracefully. A flat gap across multipliers would refute the core claim.
Standard benchmarks. We evaluate on Snow100K-L, CSD, SRRS, Rain100L/H and the all-weather composite of Snow100K, RainDrop and Outdoor-Rain, using PSNR and SSIM. We add DDN-SIRR with Brisque as a real-image check.
Robustness and cost. We sweep $\lambda$ over an order of magnitude and report how inferred $K$ and PSNR change. A flat plateau is required for the method to be usable. We also report parameters, FLOPs and wall-clock latency against the 0.17-second M2Restore figure at 224×224, and three seeds per configuration.
References
- MSP-Former: Multi-Scale Projection Transformer for Single Image Desnowing
- Revitalizing Real Image Deraining via a Generic Paradigm towards Multiple Rainy Patterns
- Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Experts
- Efficient Deweahter Mixture-of-Experts with Uncertainty-Aware Feature-Wise Linear Modulation
- M2Restore: Mixture-of-Experts-Based Mamba-CNN Fusion Framework for All-in-One Image Restoration
- DirMoE: Dirichlet-Routed Mixture of Experts
- Blind Image Denoising via Dependent Dirichlet Process Tree
- GRIDS: Grouped Multiple-Degradation Restoration with Image Degradation Similarity
- DP-space: Bayesian Nonparametric Subspace Clustering with Small-variance Asymptotics
- PromptRestorer: A Prompting Image Restoration Method with Degradation Perception
- A Bayesian Nonparametric Framework For Learning Disentangled Representations
- A Bayesian Nonparametric Approach to Image Super-Resolution.
- SmartAssign: Learning A Smart Knowledge Assignment Strategy for Deraining and Desnowing