SASFormer: Stage-Aware Selective Attention Transformer for All-in-One Blind Image Restoration

Xiangpeng Tian, Mingyang Li, Fajing Liu, Xiaofeng Wang, Xiaohai He, Chao RenarXiv 2026 · DOI · arXiv

TL;DR

SASFormer is a new "all-in-one" image restoration model that uses a single architecture to fix multiple types of image damage (like noise, rain, and haze) without needing to know the degradation type beforehand. Its core innovation is a stage-aware design that uses standard hybrid attention in the encoder to preserve image details and novel adaptive selective attention in the decoder to dynamically filter out degradation-specific artifacts.

What Problem Does This Paper Address?

Image restoration—removing noise, rain, haze, or blur—is traditionally handled by task-specific models. "All-in-one" restoration aims for a "Swiss Army knife" approach: one model to rule them all.

However, existing all-in-one models often treat the entire network uniformly, applying the same attention mechanisms from start to finish. The authors argue this is suboptimal. The encoder needs to be a "preservationist," building robust representations that keep both content and degradation cues intact. The decoder, conversely, needs to be a "perfectionist," selectively filtering those features to reconstruct a clean image. Using static, fixed attention across both stages fails to account for how different degradations (e.g., local rain streaks vs. global haze) require different receptive fields and channel priorities.

Methods

SASFormer (Stage-Aware Selective Attention Transformer) utilizes a U-shaped backbone but differentiates the modeling strategies between the encoding and decoding stages.

Fig. 3: Overview of the proposed SASFormer architecture. (a) The model adopts a U-shaped backbone with different Transformer blocks in the encoding and decoding stages. (b) The encoder integrates transposed channel self-attention (CSA) [15] and spatial self-attention (SSA) [32], while the decoder incorporates adaptive selective channel self-attention (ASCSA) and adaptive selective spatial self-attention (ASSSA). ASCSA comprises ACSM and AASM, with ACSM responsible for query-channel selection and AASM performing attention selection using ASVM. ASSSA employs another independent ASVM instance to predict fusion weights for spatial attention outputs at different window sizes.

1. The Encoder: Representation Preservation

The encoder uses established Hybrid Attention (combining Channel and Spatial Self-Attention). This allows the model to capture both global semantic context and local structural details, ensuring that the "raw material" for restoration is preserved as the feature maps move deeper into the network.

2. The Decoder: Restoration-Oriented Selection

The decoder introduces two novel adaptive modules to handle the "heavy lifting" of clean reconstruction:

  • Adaptive Selective Channel Self-Attention (ASCSA): Instead of using all channels equally, it uses an Adaptive Channel Selection Module (ACSM) (powered by Gumbel-Softmax for differentiable binary decisions) to pick the most relevant channel subsets. It then uses an Adaptive Attention Selection Module (AASM) to mask out less useful attention entries, focusing purely on restoration-relevant features.
  • Adaptive Selective Spatial Self-Attention (ASSSA): Different degradations require different window sizes. ASSSA adaptively fuses spatial attention from multiple window sizes (e.g., 4x4, 8x8, 16x16) based on the input, allowing the model to zoom in on fine-grained noise or zoom out for structural haze.

3. Implicit Prompts

Rather than requiring explicit labels (e.g., "this is a rainy image"), SASFormer uses compact implicit prompts. These are learnable tensors that provide lightweight degradation awareness, helping the selection modules decide which features to keep and which to discard.

Datasets

The model was evaluated across several rigorous settings:

  • 3-Degradation (3D): Denoising (CBSD68), Deraining (Rain100L), and Dehazing (SOTS).
  • 5-Degradation (5D): Adds Deblurring (GoPro) and Low-light enhancement (LOL-v1).
  • Composited Degradation (11D): Uses the CDD-11 dataset, which includes complex mixtures like "low-light + haze + rain."
  • Generalization: Tested on unseen datasets like Rain100H, HIDE, and DPDD to evaluate zero-shot performance.

Key Insights and Ideas

  • Encoder-Decoder Divergence: Feature analysis (using cosine similarity) revealed that in SASFormer, encoder representations for different degradations stay distinct (preserving cues), while decoder representations progressively converge toward a unified "clean" state. This confirms the stage-aware hypothesis.
  • Dynamic vs. Static: Static channel splitting (common in models like Restormer) often fails to suppress rain-related features effectively. SASFormer’s dynamic selection allows it to "turn off" degradation-heavy channels on the fly.
  • Receptive Field Adaptivity: The authors found that weak/local degradations (like light noise) favor small spatial windows, while strong/global degradations (like heavy haze) prefer larger windows. The ASSSA module automatically learns this mapping.
Fig. 2: PSNR comparison with representative methods in the three-degradation and five-degradation settings and on composited degradations.

Key Results

SASFormer achieves a superior balance between performance and efficiency:

  • State-of-the-Art Accuracy: On the 3D benchmark, it achieved an average PSNR of 32.95 dB, outperforming recent models like MEASNet and MoCE-IR.
  • Composited Mastery: On the challenging 11-degradation CDD-11 dataset, it gained 0.74 dB over R2R and 1.01 dB over MoCE-IR-S.
  • Parameter Efficiency: Despite its performance, SASFormer is relatively compact at 23.69M parameters, making it more efficient than many competitors like PromptIR (35.59M) or DA-RCOT (40.90M).

Limitations and Future Work

While highly effective, the authors note that the multi-scale spatial attention in the decoder increases memory consumption, which can be a bottleneck for very high-resolution images.

Future research directions include:

  1. Developing even more lightweight selective attention mechanisms.
  2. Extending the framework to open-set real-world degradations that weren't seen during training.
  3. Adapting the stage-aware architecture for video restoration tasks.

Summary updated

Abstract

All-in-one image restoration aims to recover high-quality images from multiple degradation types using a single unified model. Recent methods improve unified restoration with hybrid attention, prompt learning, or routing mechanisms, but generally focus on enhancing degradation adaptivity while employing similar feature modeling strategies across network stages. Consequently, the distinct representation requirements of encoding and decoding under diverse degradations remain insufficiently explored. In this paper, we propose SASFormer, a Stage-Aware Selective Attention Transformer for all-in-one blind image restoration. SASFormer follows a stage-aware design principle: the encoder uses established hybrid channel-spatial attention to build robust content-preserving and degradation-aware representations, while the decoder employs adaptive selective attention for restoration-oriented reconstruction. To address the stage-specific requirements of decoder reconstruction, we develop Adaptive Selective Channel Self-Attention (ASCSA) and Adaptive Selective Spatial Self-Attention (ASSSA), enabling the decoder to dynamically identify restoration-relevant channel responses and spatial contexts. Specifically, ASCSA performs adaptive query-channel selection and attention masking, while ASSSA adaptively integrates spatial attention across multiple window sizes. Compact implicit prompts further modulate the selection process without requiring explicit degradation labels. Extensive experiments on 3-, 5-, and 11-degradation settings demonstrate that SASFormer achieves strong restoration performance with favorable parameter efficiency. Moreover, stage-wise representation analysis and ablation studies provide empirical support for the proposed stage-aware design and the effectiveness of decoder-side adaptive selection.
BibTeX
@misc{tian2026sasformer,
  author = {Xiangpeng Tian and Mingyang Li and Fajing Liu and Xiaofeng Wang and Xiaohai He and Chao Ren},
  title = {{SASFormer: Stage-Aware Selective Attention Transformer for All-in-One Blind Image Restoration}},
  year = {2026},
  eprint = {2611.00523},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00523},
}
Figure