Models that clean up degraded photos are increasingly trained to remove several kinds of damage at once, such as rain, haze, noise and blur. The newest ones are also selective: you can tell them to remove the rain but keep the haze, or to remove noise but leave fine texture alone. Selectivity matters because full restoration can erase signal that later tasks need. Selective models learn what "rain" means from tags attached to each training image. In practice those tags come from whichever single-task dataset the image was taken from, so they are routinely incomplete. A "rain" image from a heavy-rain dataset also contains rain-induced mist. A "haze" image also carries sensor noise and compression artifacts.
We propose to treat each training image's degradation composition as a hidden quantity and to re-estimate it during training, in the style of expectation-maximization (EM): alternately guess the hidden labels from the current model, then retrain the model on those guesses. The new ingredient is the evidence used in the guessing step. Instead of trusting the restorer's own confidence, which tends to reinforce whatever mistakes it already makes, we ask a counterfactual question: if the model is told to remove only component $k$, does the degradation detector stop seeing $k$ in the output while everything else stays the same? A small hand-verified reference set anchors this loop. The claim is that correcting tag omissions this way reduces leakage, meaning the failure where "remove rain only" also strips away haze and texture. The fix should show up as better selective restoration and better downstream task accuracy, with no new architecture.
Core Claim
Seeing Through the PRISM motivates this work. It builds a compositional latent space over CLIP embeddings so that sub-mixtures like haze plus blur can be selectively inverted. It shows that selective restoration lifts microscopy segmentation mIoU from 0.475 to 0.580 compared with full restoration. However, PRISM treats mixture labels as fixed ground truth. Our hypothesis is that when tags omit components that are actually present, the model folds the missing component into the tagged one, and this is mechanically what produces leakage. Re-inferring soft composition tags should therefore produce cleaner, more separable directions in the latent space.
Two cited findings constrain the design. First, TrainRef iteratively curates hard labels into soft distributions, using a self-supervised feature space and a reference set as small as one sample per class. It shows that loops grounded only in the model's own beliefs suffer "normality pollution" at high noise. For that reason we never let the restorer's confidence alone drive the label update. Second, Learning from Noisy Labels with No Change to the Training Process and Improved Naive Bayes with Mislabeled Data supply the idea of iteratively re-estimating a noise transition matrix, which records how true labels map to observed ones. We reuse it in a source-specific form: each source dataset gets its own probability of omitting each degradation type.
Research Program
Let $x$ be a training image and let $z \in \{0,1\}^K$ indicate which of $K$ degradation types are truly present. Let $\tilde y$ be the observed tags, and let $s$ be the source dataset the image came from. The guessing step computes a soft posterior for each component:
$$q(z_k = 1) \propto \pi_k(x)\, P(\tilde y_k \mid z_k = 1, s)\, E_k(x),$$
The three factors are defined as follows. $\pi_k(x)$ is a degradation detector's prior that type $k$ is present. $P(\tilde y_k \mid z_k, s)$ is the per-source tag-noise term. It comes from a $2\times2$ table per (source, type) pair that is re-estimated after each round. $E_k(x)$ is the counterfactual removal score. To compute it, we run the current restorer with the instruction "remove only $k$". We then measure how much the detector's confidence for $k$ drops, and we subtract how much the image changes along the other types. If $k$ is absent, removing it should be close to the identity. If $k$ is present, removal should eliminate it and leave nothing else changed. This test is harder to fool than plain confidence, because a model that has wrongly merged mist into "rain" will fail the "everything else unchanged" half of the score.
The detector is trained on synthetic mixtures together with a reference set of roughly ten hand-verified images per source. The reference set also calibrates the noise tables. This is the analogue of TrainRef's minimal reference set.
The retraining step trains a PRISM-style selective restorer. The conditioning vector is the soft posterior $q$ rather than the raw tags. Per-sample losses are weighted by posterior entropy, so ambiguous images contribute less until they are resolved. Each round costs one extra inference pass over the training set. To control overhead, rounds happen every few epochs rather than every step.
The analysis has two parts. We track whether inferred compositions converge to physically sensible values; for example, Rain100H should receive substantial haze mass, consistent with the rain-accumulation fog that JORDER explicitly models, and Rain100L should receive less. We also test whether leakage falls as tags are corrected.
Baselines
The naive reference point is a selective restorer trained on the pooled data with source-dataset tags taken at face value. This is the setting the proposal claims is broken.
The strongest existing method is PRISM itself, trained on the same pool with the same tags. On its Mixed Degradations Benchmark, PRISM reports 22.08 dB PSNR against 20.84 dB for the next best diffusion model. We expect to reproduce roughly that figure when the tags are clean.
We include PromptIR, which generates degradation prompts from features in a single pass and needs no tags. It is the strongest tag-free competitor. We also include DPC-Net, which supervises a degradation-aware network with LLaVA descriptions and reports a 33.01 dB average on the three-task benchmark. It represents the "let a vision-language model write the tags" route.
Two standard noisy-label baselines complete the comparison. The first is small-loss sample filtering. The second is one-pass relabeling, in which a degradation classifier trained on synthetic data retags the pool once before training.
Two ablations isolate the key ingredient. One runs EM without $E_k$, using confidence only, which tests whether the counterfactual score is what prevents self-reinforcement. The other runs EM without the reference set.
All methods share the same backbone (where the design allows), the same pooled data, the same number of training images seen, and the same hyperparameter search budget. The EM method's extra inference passes are charged against its compute total.
Beating PRISM means two things. On the controlled benchmark, we need lower leakage and higher selective-restoration PSNR at matched full-restoration PSNR. On the real-data tests, we need equal or better downstream accuracy. Matching PRISM under clean tags while being clearly more robust under omitted tags would count as success.
Why not just…?
Why not just use a tag-free model like PromptIR? It avoids tag noise by avoiding tags altogether. It performs monolithic restoration, though, and cannot be told what to keep. PRISM's comparison on mixed degradations places it below 20.84 dB. Our selective-restoration metrics test exactly the capability it lacks. On full restoration it remains a fair competitor, and we report that honestly.
Why not just train on synthetic sub-mixtures where composition is known? PRISM already does this. The weak point is transfer: RESIDE shows that synthetic-trained dehazers that win on SOTS lose to simpler prior-based methods on real-world RTTS detection. Real paired datasets are therefore needed in the training mix, and that is where tags go wrong. The synthetic-only variant is included as a baseline on RTTS.
Why not just retag once with a classifier or vision-language model? This is the most serious threat. If a synthetic-trained detector already tags real images accurately, iteration adds nothing. BIDeN reports over 90% component identification on its Cityscapes task, which suggests one-pass tagging may be strong in-domain. The one-pass baseline uses the same detector as our prior $\pi_k$. If EM fails to beat it, especially on real sources whose degradations differ from the synthetic ones, we will conclude that the counterfactual score is unnecessary.
Why not just filter high-loss samples? Omission noise here is systematic. Essentially every Rain100H image is mis-tagged in the same way, so filtering would either discard the whole source or keep the error. The controlled benchmark measures this directly.
Proposed Experiments
Controlled benchmark. We build mixtures from BSD400/WED noise, Rain100L/Rain100H-style rain, RESIDE ITS/OTS haze and GoPro blur. Because true compositions are known, we can mimic pooling by revealing only one tag per image, with omission rates swept from 20% to 80%. We also run a condition where tags come from an actual VLM. Metrics are tag-recovery F1 and full-restoration PSNR/SSIM on BSD68, Rain100L, Rain100H and SOTS. Reference points are AirNet's 34.90 dB on Rain100L (AirNet) and JORDER-R's 23.45 dB on Rain100H. We also measure selective-restoration PSNR against synthesized sub-mixture targets. Leakage is defined as the PSNR of untouched components after a "remove only $k$" instruction.
Real-data evaluation. We train on the real pooled datasets. We evaluate dehaze-then-detect mAP on RTTS and the BID real-scenario deraining task, PRISM's mixed benchmark, and its microscopy segmentation protocol, where the target is to stay at or above its 0.580 mIoU.
Decisive checks. The proposal holds only if all of the following are true. Leakage must fall monotonically as EM rounds correct the tags. The confidence-only ablation must stall or degrade at high omission rates; if it matches full EM, the counterfactual term is unjustified. EM must beat one-pass relabeling on RTTS. If one-pass relabeling ties EM on the real data, we will report that a single good tagger suffices and that the iterative loop is unnecessary.
References
- Seeing Through the PRISM: Compound & Controllable Restoration of Scientific Images
- TrainRef: Curating Data with Label Distribution and Minimal Reference for Accurate Prediction and Reliable Confidence
- Learning from Noisy Labels with No Change to the Training Process
- Improved Naive Bayes with mislabeled data
- Deep Joint Rain Detection and Removal from a Single Image
- PromptIR: Prompting for All-in-One Image Restoration
- DPC-Net: Dual-Prior Collaborative Network for All-in-One Image Restoration
- RESIDE: A Benchmark for Single Image Dehazing
- Blind Image Decomposition
- All-In-One Image Restoration for Unknown Corruption