Image restoration networks remove rain, haze, darkness and noise from photographs. A growing family of them accepts a short text description of the damage ("rain and low light") and uses it to choose which repairs to make. Every such system in the literature treats that description as one global signal for the whole image. Real damage is rarely uniform, though. Haze thickens with distance, rain covers the sky but not a covered porch, and a streetlight brightens one corner of a dark scene. A global prompt therefore pushes the network to "dehaze" pixels that were never hazy. There is a second problem: nobody has checked whether these networks actually read the prompt, or whether they quietly ignore it and rely on the image alone.
We propose that every image patch should consult the individual words and phrases of a description, through a sparse attention step that lets a patch attend to only a few words. A learned per-patch gate then decides how much of what was retrieved to mix in. The gate is meant to open where the pixels give weak evidence about the damage and to stay shut where they are clear. We test this on a benchmark of spatially mixed degradations built from existing composite-degradation data. We use counterfactual prompt swaps to measure whether the text is used at all, and we state in advance the conditions under which a much simpler class-label conditioning should do just as well.
Core Claim
Text-guided all-in-one restoration currently injects language as a global descriptor. In OneRestore, a single scene-descriptor vector (a word embedding or a pooled visual embedding) acts as the query over image features. AllRestorer queries a memory bank of predefined degradation descriptors and rescales each degradation path for the whole image. TextPromptIR folds BERT-derived guidance into transposed attention, which mixes channels rather than spatial positions. None of the three lets a spatial location select which part of a multi-phrase description applies to it.
Our claim is narrow and testable. When degradations vary across the image and are partly ambiguous from pixels alone, three things should hold. First, patch-to-phrase attention with sparse weights and a per-patch gate should restore the referenced regions better than global-descriptor methods. Second, it should over-process the clean regions less. Third, its outputs should change in the right places when the prompt is changed. When degradation is uniform, as in most of standard CDD-11, we expect no advantage over simple label conditioning, and we treat that parity as part of the result rather than something to hide.
Research Program
The backbone is a compact transformer U-Net at the scale of OneRestore (about 6M parameters), so the comparison is about the fusion step and not model size. Descriptions are encoded by a frozen CLIP text encoder into a sequence of token embeddings. They are not pooled into one vector.
At several decoder scales we insert a fusion block. First, a short text-to-image pass lets each token attend to coarse image features. The point is that a phrase like "in the upper sky" is grounded in this particular image before patches query it. This borrows the bidirectional pattern used for frames and events in Object Tracking by Jointly Exploiting Frame and Event Domain and for spatial and spectral maps in Cross-Attention in Coupled Unmixing Nets. Second, each image patch feature $x_i$ queries the grounded tokens $T$. The result is written as
$$\tilde{x}_i = g_i \cdot \mathrm{sparsemax}\!\left(\frac{q(x_i)\,k(T)^\top}{\sqrt{d}}\right) v(T) + (1-g_i)\, x_i, \qquad g_i = \sigma\big(w^\top [x_i;\, u_i]\big).$$
Here $q, k, v$ are learned projections and $d$ is their width. Sparsemax is a softmax replacement that can assign exactly zero weight, so a patch attends to only a few tokens. $g_i \in [0,1]$ is the gate for patch $i$. $u_i$ is a small learned estimate of how unreliable the local visual evidence is, computed from the local mean and variance of the patch features, the same statistics RAM uses. $\sigma$ is the logistic function.
We take the gate-plus-sparsemax design directly from GCA. That paper used the gate to keep cross-attention from washing out the original features, which it tracked as falling cosine similarity under ungated attention. It used sparsemax to localize binding sites, with nearly 90% accuracy in one head, and reported MSE on Davis improving from 0.438 to 0.393. GCA worked on one-dimensional biological sequences. Our contribution is to move the mechanism onto two-dimensional restoration feature maps, where "localization" means locating the degradation, and to tie the gate to visual uncertainty.
The motivating evidence comes from OneRestore, which found text descriptors more stable than visual ones under heavy corruption. That finding suggests text should matter most exactly where pixels are least informative, and the gate is designed to test this.
The evaluation regime needs a new test set, which we call CDD-11-Spatial. We composite CDD-11 degradation types onto the same clean scene using region masks: random blobs, depth-like gradients, and sky/foreground splits. Each image is paired with a templated multi-phrase description ("haze toward the horizon, rain across the whole frame, dark lower half") and a paraphrased free-form version. If CDD-11 does not provide spatially aligned versions of each scene, we re-synthesize the degradations on its clean images with the dataset's own procedure. Masks give us a ground truth for where each degradation is, which makes the attention measurable in the way GCA measured binding sites.
Baselines
The naive reference point is the same backbone with no conditioning at all, trained identically. It tells us how much any guidance helps on spatially mixed damage.
The strongest existing methods on composite degradation are AllRestorer and OneRestore. On CDD-11, AllRestorer reports 33.72 dB PSNR and 0.9436 SSIM, and OneRestore reports 28.72 dB with 5.98M parameters. We retrain both on the CDD-11-Spatial training split with their published architectures. They receive the same descriptions, reduced to their expected input format: degradation-class embeddings for AllRestorer's memory bank and a global word embedding for OneRestore. Every method gets the same training schedule and hyperparameter search budget. Beating them means two things. On standard CDD-11, we should stay within a few tenths of a dB of AllRestorer's 33.72 dB. On CDD-11-Spatial, we need a clear gain in PSNR measured inside the degraded regions, together with lower error inside the clean regions.
The image-only competitor is RAM. It uses no text and reports 0.45 dB over PromptIR with 6.29M parameters and 19G FLOPs, so it is the right check on whether language is needed at all. We also include PromptIR as the learned-prompt representative.
Three ablations isolate the key ingredient. Swapping sparsemax for softmax tests sparsity. Removing the gate, or replacing the per-patch gate with one global gate, tests local gating. Pooling the text into one vector while keeping everything else fixed tests whether token-level phrases matter.
We add one oracle upper bound: the same backbone given the ground-truth degradation masks as extra input channels. It shows how much of the possible spatial benefit the text route recovers.
Why not just…?
Why not just condition on a degradation label with feature-wise scaling, which is far simpler? This is the real threat. On uniform composites the text carries no more than a class label, so label-based scaling should match us there, and we expect it to. The proposal only claims an advantage when the label is spatially wrong for parts of the image. We test this directly with a label-scaling baseline whose scale is generated from the multi-hot set of degradations present. If it matches our model on region-wise PSNR in CDD-11-Spatial, the phrase-level attention is not earning its cost and we will report that.
Why not just let an image-only network find the degradation itself, as RAM does? Rain streaks are visually distinctive, and for those RAM may well localize them unaided. The case for text rests on overlapping, ambiguous mixes. OneRestore and AllRestorer both report that purely visual descriptors become unreliable under composite corruption. RAM itself reports trouble when one degradation dominates training. We split the test results by visual ambiguity, using cases such as haze over low light against isolated rain. Text should help only in the ambiguous group. If it does not, the image-only route wins.
Why not just add an unreliability-aware gate to OneRestore's existing global-descriptor attention? That design would fix feature washing but not location. A global query cannot tell the upper sky from the porch. The pooled-text ablation is essentially this model, and the comparison will show whether token-level grounding adds anything beyond the gate.
Decisive Experiments
The first check is accuracy and over-processing. We report PSNR and SSIM on standard CDD-11 and on CDD-11-Spatial, computed separately inside the degraded regions and inside the clean regions defined by the masks.
The second check is localization. For each phrase, we measure the fraction of its attention mass that falls inside the matching mask, the analogue of GCA's binding-site accuracy. We report this against the softmax ablation.
The third check is whether the text is used, adapting the counterfactual probing of C$^3$PO to restoration. We swap one phrase's location ("haze on the left" becomes "haze on the right") or its type. A model that uses the text should change its output mostly inside the newly referenced region. A model that ignores the text will barely change. A model that uses the text only globally will change everywhere. We compute this output-change ratio for every baseline, which also answers whether existing text-guided restorers read their prompts at all.
The fourth check is whether the gate behaves as intended. We correlate $g_i$ with the per-patch residual error of the image-only model. The gate should open where image-only restoration fails.
Finally, on real hazy images in RESIDE RTTS (RESIDE), descriptions come from a captioner. We report detection mAP with a fixed off-the-shelf detector, because RESIDE showed that synthetic PSNR leaders can lose on detection. We also report the no-reference scores used by WResVLM, which reports NIMA 5.084, MUSIQ 59.34 and LIQE 2.640 on RTTS. No reported mAP number was found on Lacuna for RTTS, so all detection comparisons are run by us.
The result that would sink the idea is any of three: label scaling matching us on region-wise PSNR, counterfactual swaps producing global rather than local changes, or the gate showing no relation to visual unreliability.
References
- OneRestore: A Universal Restoration Framework for Composite Degradation
- AllRestorer: All-in-One Transformer for Image Restoration under Composite Degradations
- Textual prompt guided image restoration
- Object Tracking by Jointly Exploiting Frame and Event Domain
- Cross-Attention in Coupled Unmixing Nets for Unsupervised Hyperspectral Super-Resolution
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- Restore Anything Model via Efficient Degradation Adaptation
- PromptIR: Prompting for All-in-One Image Restoration
- C$^3$PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models
- RESIDE: A Benchmark for Single Image Dehazing
- Towards Real-World Adverse Weather Image Restoration: Enhancing Clearness and Semantics with Vision-Language Models