Pyramid Architecture Search for Real-Time Image Deblurring

Xiaobin Hu, Wenqi Ren, Kaicheng Yu, Kaihao Zhang, Xiaochun Cao, Wei Liu, Bjoern H. Menze

Abstract

This paper presents PyNAS (Pyramid Neural Architecture Search), a framework designed to automate the creation of high-performance, real-time image deblurring networks. By moving away from handcrafted designs, the authors achieve state-of-the-art (SOTA) results while maintaining the speed necessary for processing 720p video at over 50 frames per second. PyNAS is the first neural architecture search framework specifically designed to optimize multi-scale and multi-patch structures for image deblurring.

Introduction

Motion blur in images, typically caused by camera shake or fast-moving objects, is "non-uniform," meaning different regions of the image exhibit varying blur patterns. To address this, researchers commonly employ multi-scale (processing at different resolutions) or multi-patch (dividing the image into smaller chunks) architectures. However, these architectures are almost always designed manually, leading to two major issues: inefficiency due to redundant layers and suboptimal configurations of scales and patches. PyNAS addresses these challenges by automating the design process to find the optimal trade-off between speed and quality.

Methodology

The authors propose a gradient-based Neural Architecture Search (NAS) tailored for the "coarse-to-fine" deblurring strategy. The search space is divided into two levels:

  • Top-Level (Pyramid Structure): The algorithm searches for the optimal number of scales (depth) and the patch hierarchy (how many patches to divide the image into at each scale).
  • Bottom-Level (Cell Operators): Within each scale, it searches for the best operations, choosing from candidates like $3\times3$ and $5\times5$ convolutions, dilated convolutions, and separable convolutions.

To ensure computational feasibility, PyNAS employs Path Binarization. Instead of keeping all possible paths active in memory, binary gates activate only one path during the search, significantly reducing the GPU memory footprint.

The PyNAS search space includes both the pyramid scale hierarchy and the specific cell operators for encoders and decoders.

Key Insights and Discovery

The automated search yielded several non-intuitive findings that challenge traditional manual design choices:

  • The "1-3-9" Discovery: The NAS found that a patch hierarchy of 1, 3, and 9 patches performed better and faster than the traditional "powers of two" (1, 2, 4, 8) hierarchy.
  • Receptive Field Evolution: Coarser scales benefit from larger kernels and dilated convolutions to capture global blur, while finer scales prefer smaller kernels for local refinement.
  • Optimal Depth: The algorithm converged on a relatively shallow scale depth ($d=3$), proving that massive, deep networks are not required if the multi-patch strategy is optimized.

Results

PyNAS was evaluated on the GoPro Dataset and the VideoDeblurring Dataset. It outperforms previous handcrafted and NAS-based models in both quality and speed.

  • Performance: The "PyNAS_d" model achieved a Peak Signal-to-Noise Ratio (PSNR) of 30.62 dB on the GoPro dataset, surpassing previous SOTA models.
  • Speed: Inference for a 720p image takes only 17ms (~58 FPS), which is well within the 30ms threshold required for real-time 30 FPS video and roughly 40x faster than some high-performance predecessors.
Comparison of PSNR vs. Runtime. PyNAS models (red stars) achieve superior quality while remaining firmly in the real-time zone.

Conclusion

PyNAS demonstrates that automated architecture search can produce highly efficient, real-time models for complex low-level vision tasks like image deblurring. While the search process is computationally intensive (requiring ~36 GPU hours), the resulting lightweight models offer a superior balance of performance and efficiency. Future work includes extending this pyramid search to other tasks such as super-resolution and dehazing.

Summary updated

Abstract

Multi-scale and multi-patch deep models have been shown effective in removing blurs of dynamic scenes. However, these methods still suffer from one major obstacle: manually designing a lightweight and high-efficiency network is challenging and time-consuming. To tackle this obstacle, we propose a novel deblurring method, dubbed PyNAS (pyramid neural architecture search network), towards automatically designing hyper-parameters including the scales, patches, and standard cell operators. The proposed PyNAS adopts gradient-based search strategies and innovatively searches the hierarchy patch and scale scheme not limited to cell searching. Specifically, we introduce a hierarchical search strategy tailored to the multi-scale and multi-patch deblurring task. The strategy follows the principle that the first distinguishes between the top-level (pyramid-scales and pyramid-patches) and bottom-level variables (cell operators) and then searches multi-scale variables using the top-to-bottom principle. During the search stage, PyNAS employs an early stopping strategy to avoid the collapse and computational issues. Furthermore, we use a path-level binarization mechanism for multi-scale cell searching to save the memory consumption. Our primary contribution is a real-time deblurring algorithm (around 58 fps) for 720p images while achieves state-of-the-art deblurring performance on the GoPro and Video Deblurring datasets.
BibTeX
@inproceedings{hu2021pyramid,
  author = {Xiaobin Hu and Wenqi Ren and Kaicheng Yu and Kaihao Zhang and Xiaochun Cao and Wei Liu and Bjoern H. Menze},
  title = {{Pyramid Architecture Search for Real-Time Image Deblurring}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00426},
}
Figure