Abstract
This paper presents PyNAS (Pyramid Neural Architecture Search), a framework designed to automate the creation of high-performance, real-time image deblurring networks. By moving away from handcrafted designs, the authors achieve state-of-the-art (SOTA) results while maintaining the speed necessary for processing 720p video at over 50 frames per second. PyNAS is the first neural architecture search framework specifically designed to optimize multi-scale and multi-patch structures for image deblurring.
Introduction
Motion blur in images, typically caused by camera shake or fast-moving objects, is "non-uniform," meaning different regions of the image exhibit varying blur patterns. To address this, researchers commonly employ multi-scale (processing at different resolutions) or multi-patch (dividing the image into smaller chunks) architectures. However, these architectures are almost always designed manually, leading to two major issues: inefficiency due to redundant layers and suboptimal configurations of scales and patches. PyNAS addresses these challenges by automating the design process to find the optimal trade-off between speed and quality.
Methodology
The authors propose a gradient-based Neural Architecture Search (NAS) tailored for the "coarse-to-fine" deblurring strategy. The search space is divided into two levels:
- Top-Level (Pyramid Structure): The algorithm searches for the optimal number of scales (depth) and the patch hierarchy (how many patches to divide the image into at each scale).
- Bottom-Level (Cell Operators): Within each scale, it searches for the best operations, choosing from candidates like $3\times3$ and $5\times5$ convolutions, dilated convolutions, and separable convolutions.
To ensure computational feasibility, PyNAS employs Path Binarization. Instead of keeping all possible paths active in memory, binary gates activate only one path during the search, significantly reducing the GPU memory footprint.
Key Insights and Discovery
The automated search yielded several non-intuitive findings that challenge traditional manual design choices:
- The "1-3-9" Discovery: The NAS found that a patch hierarchy of 1, 3, and 9 patches performed better and faster than the traditional "powers of two" (1, 2, 4, 8) hierarchy.
- Receptive Field Evolution: Coarser scales benefit from larger kernels and dilated convolutions to capture global blur, while finer scales prefer smaller kernels for local refinement.
- Optimal Depth: The algorithm converged on a relatively shallow scale depth ($d=3$), proving that massive, deep networks are not required if the multi-patch strategy is optimized.
Results
PyNAS was evaluated on the GoPro Dataset and the VideoDeblurring Dataset. It outperforms previous handcrafted and NAS-based models in both quality and speed.
- Performance: The "PyNAS_d" model achieved a Peak Signal-to-Noise Ratio (PSNR) of 30.62 dB on the GoPro dataset, surpassing previous SOTA models.
- Speed: Inference for a 720p image takes only 17ms (~58 FPS), which is well within the 30ms threshold required for real-time 30 FPS video and roughly 40x faster than some high-performance predecessors.
Conclusion
PyNAS demonstrates that automated architecture search can produce highly efficient, real-time models for complex low-level vision tasks like image deblurring. While the search process is computationally intensive (requiring ~36 GPU hours), the resulting lightweight models offer a superior balance of performance and efficiency. Future work includes extending this pyramid search to other tasks such as super-resolution and dehazing.
Summary updated