Joint Stereo Video Deblurring, Scene Flow Estimation and Moving Object Segmentation

Liyuan Pan, Yuchao Dai, Miaomiao Liu, Fatih Porikli, Quan Pan

TL;DR

This paper introduces a unified framework that simultaneously tackles three notoriously difficult computer vision tasks: deblurring stereo video, estimating 3D scene flow, and segmenting moving objects. By treating these as a single, joint optimization problem, the authors break the "chicken-and-egg" cycle where deblurring requires accurate motion data, but motion estimation requires sharp images.

What Problem Does This Paper Address?

When a stereo camera moves through a dynamic environment (like a self-driving car on a busy street), the resulting video often suffers from motion blur. This blur is a nightmare for standard computer vision algorithms.

The core challenge is a circular dependency:

  1. To deblur an image, you need to know exactly how pixels moved during the exposure (the motion kernel).
  2. To estimate motion (scene flow), you need sharp images to find matching features between frames.
  3. To segment objects, you need to distinguish between the camera's ego-motion and the independent motion of other cars or pedestrians—which is nearly impossible if the boundaries are blurred.

Traditionally, researchers have tried to solve these one by one. This paper argues that because these tasks are intrinsically linked by the underlying 3D geometry and motion of the scene, they should be solved together.

Methods

The authors propose a unified energy minimization framework. Instead of treating pixels as independent dots, they represent the world using a piece-wise planar model. Imagine the scene as a collection of small, flat tiles (superpixels) that move rigidly in 3D space.

The Unified Framework

The model's energy function consists of three main components:

  • Data Term: Ensures the deblurred images are consistent across stereo views and time (brightness constancy) and matches the observed blurred input when re-blurred.
  • Smoothness Term: Encourages neighboring "tiles" to have similar 3D orientations and motions, especially if they belong to the same object.
  • Regularization Term: Uses "Total Variation" to keep the recovered sharp images clean and free of noise.

The Secret Sauce: Semantic Priors

A key innovation is the use of semantic segmentation (via a pre-trained ResNet38). By identifying which pixels are "road," "car," or "sky" at the start, the algorithm gets a massive head start. It knows that "road" pixels should only move according to the camera's motion, while "car" pixels might have their own independent 3D velocity.

Iterative Optimization

The problem is solved in a loop:

  1. Fix the image, solve for motion: Use the current best guess of the sharp image to estimate 3D scene flow and object segments.
  2. Fix the motion, solve for the image: Use the estimated motion kernels to perform non-uniform deblurring.
  3. Repeat: Usually, just three iterations are enough to reach a high-quality result.

Datasets

The researchers evaluated their approach on:

  • KITTI Dataset: A standard for autonomous driving. They created two synthetic versions, BlurData-1 and BlurData-2, by applying realistic motion blur to sharp sequences to provide a ground-truth for comparison.
  • Synthetic Chair Sequence: A standard benchmark for stereo deblurring.
  • Real-world Stereo Video: Publicly available datasets featuring fast-moving objects and uncontrolled lighting.

Key Insights and Ideas

  • Geometry as a Constraint: By using stereo video, the authors can exploit 3D depth. Depth is a powerful "tie-breaker" for deblurring; if you know how far away an object is, you have a much better guess at its motion-induced blur.
  • Breaking the Ringing Artifacts: One of the biggest issues in deblurring is "ringing" (ghostly ripples near edges). By incorporating object segmentation into the loop, the algorithm knows exactly where an object ends and the background begins, allowing it to handle motion boundaries much more cleanly.
  • Simplicity in Complexity: While the math is complex, the underlying assumption—that the world is made of moving flat planes—is simple enough to make the optimization computationally feasible while remaining flexible enough to model complex urban scenes.

Key Results

The "joint" approach consistently outperformed state-of-the-art methods that solve these tasks separately:

  • Deblurring Quality: On the BlurData-1 (KITTI) set, the method achieved a PSNR of ~30.50 dB, outperforming previous stereo deblurring methods by roughly 1-2 dB.
  • Scene Flow Accuracy: The error rate for optical flow was reduced to 9.83%, a significant improvement over the ~13.6% seen in previous top-tier methods like PRSF.
  • Segmentation: The method achieved an F-measure of 0.7426 for moving object segmentation, significantly higher than baseline methods (which hovered around 0.60-0.67).

Qualitatively, the results show much sharper edges on moving vehicles and fewer "smearing" artifacts compared to monocular or non-joint stereo methods.

Limitations and Future Work

Despite the impressive results, the authors note several hurdles:

  • Hardware Requirements: The system requires calibrated stereo cameras, which aren't always available in consumer setups.
  • Texture-less Scenes: Like most flow-based methods, it struggles in areas with no visual detail (e.g., a flat white wall or a clear sky).
  • Complex Optics: The model currently ignores defocus blur (blur from being out of focus) and struggles with transparent or translucent objects.
  • Computational Cost: While optimized, the MATLAB/C++ implementation takes about 35 minutes to process a single scene (6 images), making it far from real-time for now.

The authors suggest that extending the method to handle longer video sequences (more than 3 frames) could further improve stability and deblurring performance.

Summary updated

Abstract

Stereo videos for the dynamic scenes often show unpleasant blurred effects due to the camera motion and the multiple moving objects with large depth variations. Given consecutive blurred stereo video frames, we aim to recover the latent clean images, estimate the 3D scene flow and segment the multiple moving objects. These three tasks have been previously addressed separately, which fail to exploit the internal connections among these tasks and cannot achieve optimality. In this paper, we propose to jointly solve these three tasks in a unified framework by exploiting their intrinsic connections. To this end, we represent the dynamic scenes with the piece-wise planar model, which exploits the local structure of the scene and expresses various dynamic scenes. Under our model, these three tasks are naturally connected and expressed as the parameter estimation of 3D scene structure and camera motion (structure and motion for the dynamic scenes). By exploiting the blur model constraint, the moving objects and the 3D scene structure, we reach an energy minimization formulation for joint deblurring, scene flow and segmentation. We evaluate our approach extensively on both synthetic datasets and publicly available real datasets with fast-moving objects, camera motion, uncontrolled lighting conditions and shadows. Experimental results demonstrate that our method can achieve significant improvement in stereo video deblurring, scene flow estimation and moving object segmentation, over state-of-the-art methods.
BibTeX
@article{pan2020joint,
  author = {Liyuan Pan and Yuchao Dai and Miaomiao Liu and Fatih Porikli and Quan Pan},
  title = {{Joint Stereo Video Deblurring, Scene Flow Estimation and Moving Object Segmentation}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=QnfmqlDUXU},
}

Loading related content...

Figure