Rethinking Coarse-to-Fine Approach in Single Image Deblurring

Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, Sung-Jea Ko

Abstract

Single-image deblurring is a fundamental task in computer vision, traditionally addressed through computationally intensive "coarse-to-fine" stacked networks. We present MIMO-UNet (Multi-Input Multi-Output U-Net), which achieves state-of-the-art performance within a single U-shaped network. By integrating multi-scale inputs and outputs with asymmetric feature fusion, MIMO-UNet provides a significant speedup—up to 40x faster than some competitors—while maintaining high restoration quality.

Introduction

Motion blur is a common degradation in digital photography, caused by camera shake or fast-moving objects. Recovering a sharp image from a single blurry input is an ill-posed problem because the blur kernel is unknown and varies across the image. To handle blur at different scales, researchers have historically favored a coarse-to-fine strategy. This approach involves a hierarchy of networks where each level processes the image at a different resolution, passing its output to the next finer level.

While this hierarchical design effectively captures multi-scale information, it introduces significant overhead. Stacking multiple sub-networks increases the number of parameters, memory consumption, and inference time, making such models impractical for deployment on edge devices or in high-speed scenarios like autonomous driving.

Methods: The MIMO-UNet Architecture

The proposed MIMO-UNet architecture rethinks the coarse-to-fine approach by embedding multi-scale processing directly into a single U-Net. This design avoids the redundancy of stacked networks while preserving the benefits of multi-scale context.

Multi-Input Single Encoder (MISE)

Standard encoders take a single high-resolution image. In contrast, MISE accepts downsampled versions of the blurry image at different stages of the encoding path. This allows the network to extract features at multiple scales simultaneously, ensuring that coarse-scale blur information is captured early.

Multi-Output Single Decoder (MOSD)

To mimic the progressive refinement of stacked networks, the MOSD generates deblurred images at multiple resolutions from different layers of the decoder. This multi-scale output provides intermediate supervision during training, guiding the network to learn effective restoration at every resolution level.

The MIMO-UNet architecture (b) utilizes multi-scale inputs and outputs within a single U-shaped network to mimic coarse-to-fine refinement efficiently, compared to traditional stacked networks (a).

Asymmetric Feature Fusion (AFF)

Traditional U-Nets use symmetric skip connections to pass features from the encoder to the decoder. MIMO-UNet introduces Asymmetric Feature Fusion (AFF), which enables more flexible information flow. AFF allows features from different scales to be fused non-linearly, facilitating the exchange of high-level semantic information and low-level spatial details across the network.

Multi-Scale Frequency Reconstruction (MSFR) Loss

Beyond standard pixel-wise loss functions, the authors introduce MSFR loss. This loss function operates in the frequency domain using Fast Fourier Transforms (FFT). Since blur is essentially a loss of high-frequency information, MSFR encourages the network to specifically recover the sharp edges and fine textures that are often smoothed out by traditional training objectives.

Experimental Results

The performance of MIMO-UNet was evaluated on the GoPro and RealBlur datasets, which are standard benchmarks for motion deblurring.

Accuracy and Speed

MIMO-UNet achieves a remarkable balance between restoration quality and computational efficiency. On the GoPro dataset, the MIMO-UNet++ variant reached a Peak Signal-to-Noise Ratio (PSNR) of 32.68 dB, surpassing previous state-of-the-art models like MPRNet.

More impressively, the base MIMO-UNet model is exceptionally fast, with an inference time of approximately 0.008 seconds (125 FPS). This is significantly faster than competitors like SRN (1.87s) or MPRNet (0.18s).

Comparison of PSNR vs. Runtime on the GoPro dataset. MIMO-UNet variants (red stars) occupy the top-left region, demonstrating superior accuracy and significantly lower runtime compared to previous SOTA methods.

Downstream Task Performance

To demonstrate real-world utility, deblurred images were passed through a pre-trained object detector (PFPNet). While the detector failed on blurry images, the restoration provided by MIMO-UNet++ allowed it to accurately identify cars and pedestrians, highlighting the model's potential for improving autonomous system reliability.

Conclusion

MIMO-UNet demonstrates that the physical stacking of networks is not a prerequisite for effective multi-scale deblurring. By integrating multi-scale inputs, outputs, and asymmetric fusion into a single U-Net, the authors have created a highly efficient architecture that sets a new standard for the speed-accuracy trade-off in image restoration. Future work may extend this efficient multi-scale framework to other tasks such as image dehazing and super-resolution.

Summary updated

Abstract

Coarse-to-fine strategies have been extensively used for the architecture design of single image deblurring networks. Conventional methods typically stack sub-networks with multi-scale input images and gradually improve sharpness of images from the bottom sub-network to the top sub-network, yielding inevitably high computational costs. Toward a fast and accurate deblurring network design, we revisit the coarse-to-fine strategy and present a multi-input multi-output U-net (MIMO-UNet). The MIMO-UNet has three distinct features. First, the single encoder of the MIMO-UNet takes multi-scale input images to ease the difficulty of training. Second, the single decoder of the MIMO-UNet outputs multiple deblurred images with different scales to mimic multi-cascaded U-nets using a single U-shaped network. Last, asymmetric feature fusion is introduced to merge multi-scale features in an efficient manner. Extensive experiments on the GoPro and RealBlur datasets demonstrate that the proposed network outperforms the state-of-the-art methods in terms of both accuracy and computational complexity. Source code is available for research purposes at https://github.com/chosj95/MIMO-UNet.
BibTeX
@misc{cho2021rethinking,
  author = {Sung-Jin Cho and Seo-Won Ji and Jun-Pyo Hong and Seung-Won Jung and Sung-Jea Ko},
  title = {{Rethinking Coarse-to-Fine Approach in Single Image Deblurring}},
  year = {2021},
  eprint = {2108.05054},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.05054},
}
Figure