High-fidelity image restoration tasks—such as super-resolution, denoising, deblurring, and deraining—are inherently computationally expensive. To restore fine details, standard deep learning models typically rely on massive convolutional neural networks (CNNs) or dense self-attention mechanisms. While these architectures achieve state-of-the-art results on high-end GPUs, deploying them locally on resource-constrained edge hardware (such as smartphones, drones, smart TVs, or autonomous vehicle sensors) introduces severe bottlenecks in memory bandwidth, latency, and power consumption.
To bridge this gap, a major research direction focuses on redesigning image restoration pipelines to minimize computational overhead. Rather than simply shrinking existing models, researchers are rethinking the mathematical primitives of image processing, exploiting spatial redundancies in visual data, and replacing expensive floating-point operations with highly efficient hardware-native alternatives.
Rethinking the Computational Core
Traditional deep networks rely on deep stacks of spatially invariant convolutions and complex non-linear activation functions. To make these models suitable for edge hardware, researchers are stripping away unnecessary operations and adopting alternative mathematical frameworks that offer better performance-to-compute ratios.
For example, Liangyu Chen et al. challenge the necessity of traditional activation functions in Simple Baselines for Image Restoration. They demonstrate that non-linear activations (like ReLU or Sigmoid) can be completely omitted or replaced with simple multiplication operations. Their resulting architecture, NAFNet, drastically reduces multiply-accumulate operations while maintaining world-class denoising and deblurring performance.
Traditional Block: [Conv] -> [ReLU] -> [Conv] -> [ReLU] -> Output
NAFNet Block: [Conv] ------> [Gated Linear Unit] ----> Output (No Activation Functions)
Instead of scaling up CNN depth to capture wide spatial contexts, other approaches utilize recurrent or invertible structures to save memory:
- Spatially Variant RNNs: jiawei zhang et al. introduce a spatially variant Recurrent Neural Network in Dynamic Scene Deblurring Using Spatially Variant Recurrent Neural Networks. By learning unique weights for every pixel and processing images iteratively rather than through massive parallel layers, the authors slash model size and inference time by roughly 10x.
- Invertible Networks: Yang Liu 0249 leverages Invertible Neural Networks (INNs) in Invertible Denoising Network: A Light Solution for Real Noise Removal. Because INNs can reconstruct inputs from outputs, they eliminate the need to store intermediate feature maps during processing, offering a lightweight, memory-efficient alternative for real-world noise removal.
- Spiking Neural Networks: Moving closer to hardware-level efficiency, Tianyu Song et al. propose ESDNet in Learning a Spiking Neural Network for Efficient Image Deraining. By utilizing bio-inspired Spiking Neural Networks (SNNs), the model communicates via discrete, event-driven binary spikes rather than continuous floating-point values, cutting energy consumption by 54%.
Content-Adaptive and Sparse Processing
A major source of inefficiency in standard image processing is uniform computation: treating every pixel in an image with the same amount of mathematical rigor. In reality, a flat sky or a smooth wall requires far less computational effort to upscale or denoise than a highly textured region or a complex edge.
[Input Image] ---> [Mask Generator] ---> Flat Regions ---> (Low Compute / Global Pass)
---> Sharp Edges ---> (High Compute / Local Refinement)
To exploit this spatial redundancy, Xiaotao Hu et al. introduce a "Restore Globally, Refine Locally" framework in Restore Globally, Refine Locally: A Mask-Guided Scheme to Accelerate Super-Resolution Networks. The model uses a lightweight base network to process the entire image globally, while a mask-guided routing mechanism identifies complex, high-frequency patches for localized, high-resolution refinement. This prevents the network from wasting cycles on easy-to-restore flat regions.
This adaptive philosophy has been successfully applied to network routing and attention mechanisms:
- Dynamic Path Selection: In Path-Adaptive Matting for Efficient Inference Under Various Computational Cost Constraints, Qinglin Liu et al. propose a framework that dynamically alters its internal network path based on the complexity of the input image and the immediate hardware FLOP constraints of the device.
- Sparse Attention: Vision Transformers (ViTs) are highly effective at capturing global context but suffer from quadratic computational complexity relative to image size. To mitigate this, Xiang Chen et al. develop a learnable Top-k Sparse Attention mechanism in Learning A Sparse Transformer Network for Effective Image Deraining. By filtering out irrelevant background features and focusing attention matrices only on critical degraded regions, they preserve the expressive power of transformers within a fraction of the computational budget.
Binarization, Quantization, and Look-Up Tables
Beyond architectural tweaks, a powerful way to optimize models for edge deployment is to change how numbers are represented and calculated. Standard deep networks operate on 32-bit floating-point numbers (FP32), which require power-hungry hardware multipliers.
Cheeun Hong addresses this in CADyQ: Content-Aware Dynamic Quantization for Image Super-Resolution by introducing a content-aware dynamic quantization strategy. Instead of quantizing the entire network uniformly, CADyQ adaptively assigns different bit-widths to different image patches and network layers. Complex textures receive higher bit-widths to preserve fidelity, while flat regions are compressed to ultra-low bit-widths, achieving a 95% reduction in total Bit Operations (BitOPs).
Taking quantization to its absolute limit, binarization compresses weights and activations down to 1-bit values (-1 or +1). This converts expensive floating-point multiplications into simple, hardware-native bitwise XNOR and popcount operations.
Standard Convolution: O = ∑ (Weight_FP32 × Activation_FP32) --> High Power, High Latency
Binarized Network: O = Popcount(Weight_1bit XNOR Activation_1bit) --> Low Power, 58x Speedup
In Binarized Neural Network for Single Image Super Resolution, Jingwei Xin et al. introduce a Bit-Accumulation Mechanism (BAM) that refines binary approximations across layers. This allows binarized networks to overcome their typical accuracy drops and achieve a 58x speedup and 32x size reduction compared to full-precision equivalents. Similarly, Yuanhao Cai et al. deploy binarization in Binarized Spectral Compressive Imaging to perform memory-efficient hyperspectral image reconstruction directly on low-power mobile sensors.
Replacing Inference with Look-Up Tables
The most radical departure from traditional neural network inference involves bypassing mathematical operations entirely via Look-Up Tables (LUTs). In a LUT-based approach, the mapping between a local patch of degraded pixels and its restored counterpart is pre-computed and stored directly in memory. During inference, the device simply fetches values from memory based on the input pixel values, consuming virtually zero computational power.
However, traditional LUTs suffer from an "exponential curse": if you want to look up a patch of $N$ pixels simultaneously, the table size grows exponentially with $N$, limiting traditional LUTs to tiny receptive fields.
Traditional LUT: [Input Patch] ----------------------------> [Single Massive Table] (Size grows exponentially)
MuLUT Structure: [Input Patch] ---> [Sub-table 1] ---\
---> [Sub-table 2] ----+----> [Deep Network of LUTs] (Large Receptive Field)
---> [Sub-table 3] ---/
To solve this, Jiacheng Li et al. introduce MuLUT in Toward DNN of LUTs: Learning Efficient Image Restoration with Multiple Look-Up Tables. The authors organize multiple parallel and sequential look-up tables into a structure that mimics a deep neural network. By factoring a large receptive field into multiple smaller, interacting tables, MuLUT breaks the exponential storage barrier. This allows edge devices like smartphones and smart TVs to perform high-quality super-resolution and denoising with a 100x increase in energy efficiency over even the most lightweight deep learning models.