TL;DR
This paper introduces EDIBNet, a novel, low-complexity neural network for depth-guided image deblurring that achieves state-of-the-art image quality while drastically reducing computational cost. By leveraging the Discrete Wavelet Transform (DWT) and efficient depth feature fusion, EDIBNet is optimized for deployment on resource-constrained edge devices like smartphones.
What Problem Does This Paper Address?
Image deblurring is a crucial task in computational photography, aiming to restore sharp images from blurry inputs. While deep learning models have achieved impressive results, their computational demands often make them impractical for real-time applications on mobile devices. Existing methods that incorporate complementary information, such as depth maps from mobile LiDAR sensors, have shown promise but still suffer from high complexity. This paper addresses the need for an efficient yet high-quality deblurring solution that can run on edge devices, bridging the gap between cutting-edge performance and practical deployment.
Methods
The core of EDIBNet lies in two key innovations:
- Discrete Wavelet Transform (DWT) for efficient representation: The input blurred image is decomposed into different frequency sub-bands (LL, LH, HL, HH) using a 2-level Haar DWT. This allows the network to process only the low-frequency LL sub-band for deblurring operations, significantly reducing computational load. High-frequency components, which contribute less to deblurring quality, are largely skipped until reconstruction. This approach effectively trades spatial resolution and channel count for computational efficiency.
- Lightweight Depth-Guided Feature Fusion: Depth information from mobile LiDAR is integrated into the network via efficient "adapter" modules. These adapters process depth features to generate "prompt signals" that adaptively modulate the image features within the decoder. This fusion emphasizes structural guidance, aiding in sharpness and boundary restoration without incurring significant computational overhead. The architecture employs a compact encoder-decoder backbone.
Datasets
The authors utilize a subset of the ARKitScenes dataset for training and validation. This dataset is specifically chosen for its availability of RGB-D image pairs with associated real depth maps produced by mobile LiDAR sensors. The training set contains 29,264 image-depth pairs, and the validation set consists of 500 pairs.
Key Insights and Ideas
- Wavelet Transform as a Computational Bottleneck Reducer: Decomposing the image into wavelet sub-bands and focusing deblurring on the LL component is a highly effective strategy for reducing computational complexity. This insight is crucial for making advanced image processing feasible on edge devices.
- Depth as a Structural Guide, Not Just a Regularizer: The paper demonstrates that depth information can be more than just a regularization term; it can actively guide the deblurring process by highlighting structural elements, leading to sharper and more accurate restorations.
- The Power of "Skipping" Information: By strategically skipping high-frequency wavelet sub-bands that have minimal impact on deblurring quality, the network achieves substantial efficiency gains. This highlights the importance of understanding information redundancy in image processing.
- Balancing Accuracy and Efficiency: The 2-level DWT decomposition is shown to strike an optimal balance between deblurring performance and computational cost, outperforming 1-level and 3-level configurations.
Key Results
EDIBNet achieves competitive image quality comparable to state-of-the-art models while reducing computational complexity by up to two orders of magnitude.
- Runtime: The model processes a high-resolution frame in approximately 0.2 seconds on an NVIDIA Jetson Orin Nano edge device. This is significantly faster than models like Restormer, which can take tens of seconds.
- Complexity: EDIBNet boasts a drastic reduction in FLOPs and memory requirements, making it suitable for edge deployment. For instance, compared to Restormer, the reduction is on the order of two magnitudes.
- Performance: The proposed EDIBNet (with 32 channels) achieves a PSNR of 35.10 dB and an SSIM of 0.9681, outperforming its own variant without depth guidance and being competitive with larger models. Integrating depth information provides an additional gain of 0.14 dB in PSNR.
Limitations and Future Work
- Limitations: While the adapter design is more efficient, its PSNR is marginally lower than a previously proposed adapter from [11]. The paper notes that for models exceeding 8GB memory limits, runtime is impacted by image tiling and sequential evaluation, indicating that even some "efficient" SOTA models are challenging to run on edge platforms.
- Future Work: The paper focuses on the 2-level Haar DWT and suggests that exploring other wavelet types or decomposition levels might yield further improvements. The authors also hint at the potential for further streamlining the adapter architecture. The current work provides a strong foundation for future research into efficient, depth-guided image restoration on edge devices.
Summary updated