TL;DR
All-in-one image restoration models struggle with severe domain gaps and cross-task performance trade-offs when deployed in real-world adverse weather. DDTNet resolves this by disentangling degradation patterns from unlabeled target-domain images and transferring them onto clean source images. This creates domain-adapted training pairs on the fly, enabling effective test-time fine-tuning that substantially boosts all-in-one de-weathering performance.
What Problem Does This Paper Address?
Unified "all-in-one" restoration models aim to remove multiple weather degradations—such as rain, snow, and haze—within a single neural network. However, they suffer from two major hurdles in practice:
- Cross-task interference: Joint multi-task training often forces models to compromise, frequently performing worse than networks tuned specifically for a single degradation.
- Domain shifts: Models trained on synthetic datasets experience steep performance drops when deployed on diverse, unpredictable real-world weather conditions.
Because real-world test scenes do not have paired ground-truth clean images, traditional supervised domain adaptation cannot be directly applied during inference or deployment.
Methods
DDTNet introduces a degradation-transfer strategy: instead of attempting unsupervised cleaning of degraded target images, it extracts the target degradation signatures and injects them into clean source images. The resulting pairs $(I_{\text{transferred}}, I_{\text{clean}})$ serve as supervised training data to fine-tune arbitrary all-in-one restoration backbones.
Target Degraded Image ──► Degradation Encoder ──► DDM (DCA) ──┐
▼
Clean Source Image ──► Content Encoder ──────────► Transfer Decoder ──► Transferred Image
The core innovation is the Degradation Disentanglement Module (DDM), powered by a two-stage Degradation Coupled Attention (DCA):
- Stage 1 (Degradation Encoding): A set of learnable degradation-sensing tokens ($M=256$) act as queries across the image feature maps to distill weather-specific and weather-general degradation features into compact degradation tokens.
- Stage 2 (Content Suppression): The image tokens then act as queries against the newly formed degradation tokens to aggregate pure degradation cues while suppressing background scene semantics.
DCA is applied across both spatial dimensions ($\text{DCA}_S$, modeling geometric structures like streaks and snowflakes) and channel dimensions ($\text{DCA}_C$, capturing contrast, veiling effects, and color attenuation). The transferred images are generated via a multi-scale Transfer Decoder and supervised using an $\ell_1$ loss on synthesized triplets.
Datasets
- Training DDTNet: 15,000 synthetic degraded–clean–mixed triplets (5,000 each for rain, snow, and haze). Rain masks were drawn from Rain100H and Rain100L, snow masks from Snow100K, and haze parameters from RESIDE.
- Target Evaluation: The real-world WeatherStream benchmark, consisting of 4,500 hazy, 3,000 rainy, and 3,960 snowy images (including complex mixed conditions like rain-with-haze and snow-with-haze) paired with clean ground truth.
- Ablation Testing: A 900-triplet synthetic test set (300 triplets per task from Rain100, Snow100K, and RESIDE).
Key Insights and Ideas
- Role-reversing attention decouples degradation: By having learnable tokens first query the image and then having image features query the distilled tokens, the network isolates weather artifacts from underlying image content without needing explicit physical scattering priors.
- Semantic clustering of degradation tokens: t-SNE analysis shows that without explicit class supervision during inference, the learned degradation tokens automatically cluster by weather condition (distinct rain, haze, and snow clusters).
- All-in-one surpasses single-task: After fine-tuning on DDTNet-generated pairs for just a single epoch, unified models not only overcome multi-task interference but actually outperform their single-task counterparts.
Key Results
Evaluating on the real-world WeatherStream benchmark demonstrated clear improvements across multiple state-of-the-art all-in-one models:
- PromptIR: Average PSNR increased from 21.13 dB to 22.22 dB (+1.09 dB gain; +1.64 dB on haze).
- AdaIR: Average PSNR increased from 20.92 dB to 22.36 dB (+1.44 dB gain; +1.90 dB on haze).
- DFPIR: Average PSNR improved from 21.19 dB to 22.13 dB (+0.94 dB gain).
- vs. Noise-DA (General Domain Adaptation): When applied to the baseline architecture used in Noise-DA, DDTNet improved PSNR from 16.74 dB to 22.41 dB (+5.67 dB), whereas Noise-DA achieved only 17.76 dB and degraded haze restoration performance.
Qualitatively, fine-tuned models effectively eliminated lingering haze veils and blended streak artifacts that baseline models failed to remove. DDTNet has 31M parameters and runs in 33 ms on an NVIDIA RTX A5000 GPU (at $256 \times 256$ resolution).
Limitations and Future Work
- Dependence on synthetic triplets: Offline training of DDTNet relies on degraded–clean–mixed triplets generated using heuristic masks and atmospheric scattering models, which may constrain scaling when realistic simulation parameters are scarce.
- Test-time compute overhead: Adapting to new distributions requires an on-the-fly fine-tuning phase (one epoch) rather than purely zero-shot inference.
Summary updated