Image restoration tasks—such as super-resolution, deblurring, and dehazing—have increasingly relied on massive vision transformers and diffusion models. While these heavy architectures achieve remarkable visual fidelity, their high parameter counts make them impractical for real-time deployment on edge devices like smartphones.
Unlike open-ended image generation, image restoration is fundamentally a reconstruction task constrained by an underlying degraded input. This structural constraint means that massive generative capacity is often redundant. Researchers are exploiting this insight to build ultra-lightweight architectures that achieve state-of-the-art restoration quality by replacing brute-force scaling with targeted architectural efficiency.
Linearizing and Localizing Attention
Standard self-attention mechanisms scale quadratically ($O(N^2)$) with the number of pixels, forcing models to either use massive parameter counts to stabilize global context or restrict their receptive fields. To bypass this bottleneck, efficient architectures rethink how tokens and attention matrices are constructed.
For instance, Fu-Jen Tsai et al. developed Stripformer: Strip Transformer for Fast Image Deblurring, which replaces traditional square patches with horizontal and vertical strip tokens. This design directly targets the directional nature of motion blur, capturing long-range context along blur trajectories without the parameter overhead of full spatial attention.
Alternatively, the quadratic complexity of the softmax function can be mathematically bypassed. In MB-TaylorFormer: Multi-branch Efficient Transformer Expanded by Taylor Formula for Image Dehazing, Kaihao Zhang et al. approximated standard softmax-attention using a first-order Taylor expansion. This mathematical substitution achieves linear computational complexity ($O(N)$) while preserving a global receptive field, allowing a highly compact model to match or exceed the performance of much larger networks.
Similarly, window-based attention mechanisms, such as those used in SwinIR by Jingyun Liang et al., restrict attention to shifted local windows, drastically reducing the parameter footprint required to model long-range spatial dependencies.
Shifting to Frequency and Structural Domains
Operating directly on raw spatial pixels forces a network to learn both high-frequency textures and low-frequency structures simultaneously, inflating model size. By transforming the input into alternative mathematical domains, networks can process distinct components of an image with specialized, lightweight modules.
Wavelet and Fourier transforms are highly effective for this purpose. In EWT: Efficient Wavelet-Transformer for Single Image Denoising, Juncheng Li et al. performed feature extraction in the wavelet domain rather than the spatial domain. Because wavelets naturally segregate an image into sub-bands of differing frequencies, the model can deploy a hybrid, dual-stream architecture that processes high- and low-frequency details separately, reducing GPU memory usage by 60% and cutting down the required parameter scale.
Another approach is to embed domain-specific physical priors directly into the architecture. Ziteng Cui et al. introduced the Illumination Adaptive Transformer, which mimics a camera's internal Image Signal Processing (ISP) pipeline. By structuring the network around explicit exposure and tone-mapping stages, the model achieves state-of-the-art enhancement using fewer than 100,000 parameters—proving that domain-specific structural priors can completely replace raw parameter scaling.
Eliminating Redundancy via Weight Sharing and Reparameterization
Deep convolutional networks often suffer from feature redundancy, where different layers or channels learn highly similar representations. Capitalizing on this redundancy allows for extreme model compression.
Axi Niu et al. demonstrated this in GRAN: Ghost Residual Attention Network for Single Image Super Resolution by integrating Ghost Modules. Instead of computing every feature map via expensive standard convolutions, Ghost Modules generate a small set of intrinsic feature maps and then apply cheap, linear operations to produce the remaining redundant features. This simple shift reduces parameters and FLOPs by over ten times compared to baseline models.
Weight sharing across time or depth offers another avenue for parameter reduction. In Recurrently-Trained Super-Resolution, Saem Park utilized a multi-stage target update strategy where a single lightweight network is recurrently retrained on its own enhanced outputs. This iterative refinement allows a model with 90% fewer parameters to match the perceptual sharpness of heavy, deep architectures.
Furthermore, structural reparameterization—as highlighted in the mobile challenge report by Andrey Ignatov et al.—allows complex multi-branch topologies used during training to be collapsed into simple, ultra-fast single-branch convolutions during inference, enabling footprints as small as 23 KB.
Decoupling Content from Degradation
When large generative models like Stable Diffusion are adapted for restoration tasks, they waste significant capacity on "imagining" semantic content. Parameter-efficient restoration frameworks isolate the generative mechanism so it only handles the corruption process, leaving content preservation to lighter structures.
In Edge-SD-SR: Low Latency and Parameter Efficient On-device Super-Resolution with Stable Diffusion via Bidirectional Conditioning, Isma Hadji et al. stripped away the high-level semantic imagination components of Stable Diffusion. Because super-resolution focuses on reconstruction rather than creative invention, the authors reduced the model size by roughly 80% without sacrificing visual fidelity.
Similarly, frameworks can decouple content from degradation explicitly. Sen Chen et al. proposed the Unsupervised Degradation Representation Aware Transform for Real-World Blind Image Super-Resolution (DRAT), which uses a dual-encoder system to isolate pure degradation information from image content. By explicitly modeling how an image was blurred or corrupted rather than relying on a massive general-purpose generative model, the architecture achieves clean, artifact-free restorations with a fraction of the parameter footprint.
To further optimize these generative pipelines, Runyi Li et al. introduced CTSR: Controllable Fidelity-Realness Trade-off Distillation for Real-World Image Super Resolution, which integrates low-rank adaptation (LoRA) into a distillation framework. This allows a one-step diffusion student model to achieve high-quality upscaling with a tiny fraction of the trainable parameters required by standard diffusion architectures.