TL;DR
SASFormer is a new "all-in-one" image restoration model that uses a single architecture to fix multiple types of image damage (like noise, rain, and haze) without needing to know the degradation type beforehand. Its core innovation is a stage-aware design that uses standard hybrid attention in the encoder to preserve image details and novel adaptive selective attention in the decoder to dynamically filter out degradation-specific artifacts.
What Problem Does This Paper Address?
Image restoration—removing noise, rain, haze, or blur—is traditionally handled by task-specific models. "All-in-one" restoration aims for a "Swiss Army knife" approach: one model to rule them all.
However, existing all-in-one models often treat the entire network uniformly, applying the same attention mechanisms from start to finish. The authors argue this is suboptimal. The encoder needs to be a "preservationist," building robust representations that keep both content and degradation cues intact. The decoder, conversely, needs to be a "perfectionist," selectively filtering those features to reconstruct a clean image. Using static, fixed attention across both stages fails to account for how different degradations (e.g., local rain streaks vs. global haze) require different receptive fields and channel priorities.
Methods
SASFormer (Stage-Aware Selective Attention Transformer) utilizes a U-shaped backbone but differentiates the modeling strategies between the encoding and decoding stages.
1. The Encoder: Representation Preservation
The encoder uses established Hybrid Attention (combining Channel and Spatial Self-Attention). This allows the model to capture both global semantic context and local structural details, ensuring that the "raw material" for restoration is preserved as the feature maps move deeper into the network.
2. The Decoder: Restoration-Oriented Selection
The decoder introduces two novel adaptive modules to handle the "heavy lifting" of clean reconstruction:
- Adaptive Selective Channel Self-Attention (ASCSA): Instead of using all channels equally, it uses an Adaptive Channel Selection Module (ACSM) (powered by Gumbel-Softmax for differentiable binary decisions) to pick the most relevant channel subsets. It then uses an Adaptive Attention Selection Module (AASM) to mask out less useful attention entries, focusing purely on restoration-relevant features.
- Adaptive Selective Spatial Self-Attention (ASSSA): Different degradations require different window sizes. ASSSA adaptively fuses spatial attention from multiple window sizes (e.g., 4x4, 8x8, 16x16) based on the input, allowing the model to zoom in on fine-grained noise or zoom out for structural haze.
3. Implicit Prompts
Rather than requiring explicit labels (e.g., "this is a rainy image"), SASFormer uses compact implicit prompts. These are learnable tensors that provide lightweight degradation awareness, helping the selection modules decide which features to keep and which to discard.
Datasets
The model was evaluated across several rigorous settings:
- 3-Degradation (3D): Denoising (CBSD68), Deraining (Rain100L), and Dehazing (SOTS).
- 5-Degradation (5D): Adds Deblurring (GoPro) and Low-light enhancement (LOL-v1).
- Composited Degradation (11D): Uses the CDD-11 dataset, which includes complex mixtures like "low-light + haze + rain."
- Generalization: Tested on unseen datasets like Rain100H, HIDE, and DPDD to evaluate zero-shot performance.
Key Insights and Ideas
- Encoder-Decoder Divergence: Feature analysis (using cosine similarity) revealed that in SASFormer, encoder representations for different degradations stay distinct (preserving cues), while decoder representations progressively converge toward a unified "clean" state. This confirms the stage-aware hypothesis.
- Dynamic vs. Static: Static channel splitting (common in models like Restormer) often fails to suppress rain-related features effectively. SASFormer’s dynamic selection allows it to "turn off" degradation-heavy channels on the fly.
- Receptive Field Adaptivity: The authors found that weak/local degradations (like light noise) favor small spatial windows, while strong/global degradations (like heavy haze) prefer larger windows. The ASSSA module automatically learns this mapping.
Key Results
SASFormer achieves a superior balance between performance and efficiency:
- State-of-the-Art Accuracy: On the 3D benchmark, it achieved an average PSNR of 32.95 dB, outperforming recent models like MEASNet and MoCE-IR.
- Composited Mastery: On the challenging 11-degradation CDD-11 dataset, it gained 0.74 dB over R2R and 1.01 dB over MoCE-IR-S.
- Parameter Efficiency: Despite its performance, SASFormer is relatively compact at 23.69M parameters, making it more efficient than many competitors like PromptIR (35.59M) or DA-RCOT (40.90M).
Limitations and Future Work
While highly effective, the authors note that the multi-scale spatial attention in the decoder increases memory consumption, which can be a bottleneck for very high-resolution images.
Future research directions include:
- Developing even more lightweight selective attention mechanisms.
- Extending the framework to open-set real-world degradations that weren't seen during training.
- Adapting the stage-aware architecture for video restoration tasks.
Summary updated