Dynamic Scene Deblurring

Tae Hyun Kim, Byeongjoo Ahn, Kyoung Mu LeeICCV 2013 · OpenReview

Abstract: Most conventional single image deblurring methods assume that the underlying scene is static and the blur is caused by only camera shake. In this paper, in contrast to this restrictive assumption, we address the deblurring problem of general dynamic scenes which contain multiple moving objects as well as camera shake. In case of dynamic scenes, moving objects and background have different blur motions, so the segmentation of the motion blur is required for deblurring each distinct blur motion accurately. Thus, we propose a novel energy model designed with the weighted sum of multiple blur data models, which estimates different motion blurs and their associated pixel-wise weights, and resulting sharp image. In this framework, the local weights are determined adaptively and get high values when the corresponding data models have high data fidelity. And, the weight information is used for the segmentation of the motion blur. Non-local regularization of weights are also incorporated to produce more reliable segmentation results. A convex optimization-based method is used for the solution of the proposed energy model. Experimental results demonstrate that our method outperforms conventional approaches in deblurring both dynamic scenes and static scenes.

The Challenge of Dynamic Scenes

Most classic image deblurring techniques operate under a restrictive assumption: the scene is static, and all blur comes from a single source, usually camera shake. In the real world, however, scenes are dynamic. If you take a photo of a moving bus while your hands are shaking, the bus has one type of blur (object motion + camera shake), while the background has another (camera shake only).

Standard "uniform" deblurring fails here because applying the bus's blur kernel to the background creates messy artifacts, and vice versa. The core challenge is a "chicken and egg" problem: to deblur the image, you need to know which pixels belong to which motion (segmentation), but to segment the motion accurately, you often need a sharp image.

Methodology: Joint Estimation and Non-Local Regularization

The authors propose a novel energy minimization model that solves for three things simultaneously: the latent sharp image ($L$), a set of multiple blur kernels ($K$), and a set of pixel-wise weight maps ($W$) that act as a "soft" segmentation.

Key technical components include:

  • Weighted Data Fidelity Term: Instead of one blur model, the data term is a weighted sum of multiple models. If a specific kernel $K_i$ explains the blur at a pixel well, its corresponding weight $W_i$ increases.
  • Non-Local Regularization for Weights: To ensure the segmentation is clean and follows object boundaries, the authors use a non-local similarity map. This looks at a large neighborhood (e.g., 80 neighbors) to enforce smoothness in the weight maps, which is much more effective than looking only at immediate neighbors.
  • Hybrid Kernel Support: The model is flexible enough to handle both simple uniform kernels and complex non-uniform kernels by representing the latter as linear combinations of basis kernels.
  • Optimization: The authors use an iterative approach, alternating between updating the sharp image, the weights, and the kernels using a first-order primal-dual algorithm.

Experimental Validation

The paper evaluates its method on a mix of synthetic and real-world data, including challenging photographs of moving buses, running bulls, and street traffic. These images are particularly difficult because they contain overlapping motion types and complex textures.

Deblurring results on a dynamic scene featuring a moving bus. (a) Input blurred image, (b-c) deblurred details, (d) full deblurred result, (e) estimated segmentation map, and (f) final output.

In the "moving bus" example shown above, the proposed method restores the text on the bus and the texture of the background, whereas previous state-of-the-art methods typically blurred the bus or ruined the background with artifacts.

Key Insights

  • Joint Estimation is Key: Estimating segmentation and blur kernels jointly allows the two processes to inform each other, leading to much cleaner boundaries.
  • Sparsity Priors for Segmentation: By using a total variation (TV) prior on the weights, the model naturally "picks" the best blur model for each region, effectively performing automatic motion segmentation.
  • Redundancy Benefits Static Scenes: Interestingly, even in static scenes with only one type of blur, using multiple "slightly different" kernels helps suppress "ringing" artifacts (halos around edges) better than a single-kernel approach.

Limitations and Future Work

While effective, the system currently requires the number of blur models ($N$) to be set manually (the authors used $N=7$). A truly "blind" system should ideally determine the number of moving objects automatically. Additionally, adding more blur models increases memory and processing requirements, suggesting that future work should focus on GPGPU implementations to make the algorithm fast enough for practical use.

Summary updated

Abstract

Most conventional single image deblurring methods assume that the underlying scene is static and the blur is caused by only camera shake. In this paper, in contrast to this restrictive assumption, we address the deblurring problem of general dynamic scenes which contain multiple moving objects as well as camera shake. In case of dynamic scenes, moving objects and background have different blur motions, so the segmentation of the motion blur is required for deblurring each distinct blur motion accurately. Thus, we propose a novel energy model designed with the weighted sum of multiple blur data models, which estimates different motion blurs and their associated pixel-wise weights, and resulting sharp image. In this framework, the local weights are determined adaptively and get high values when the corresponding data models have high data fidelity. And, the weight information is used for the segmentation of the motion blur. Non-local regularization of weights are also incorporated to produce more reliable segmentation results. A convex optimization-based method is used for the solution of the proposed energy model. Experimental results demonstrate that our method outperforms conventional approaches in deblurring both dynamic scenes and static scenes.
BibTeX
@inproceedings{kim2013dynamic,
  author = {Tae Hyun Kim and Byeongjoo Ahn and Kyoung Mu Lee},
  title = {{Dynamic Scene Deblurring}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2013},
  url = {https://openreview.net/forum?id=B1ZZ8-Gd-H},
}
Figure