Joint Stereo Video Deblurring, Scene Flow Estimation and Moving Object Segmentation

Liyuan Pan, Yuchao Dai, Miaomiao Liu, Fatih Porikli, Quan Pan

Abstract

Stereo videos of dynamic scenes often exhibit unpleasant blur effects due to camera motion and multiple moving objects with significant depth variations. This paper proposes a unified framework to jointly recover latent clean images, estimate 3D scene flow, and segment multiple moving objects from consecutive blurred stereo frames. By representing dynamic scenes with a piece-wise planar model, we exploit the intrinsic connections between these tasks, expressing them as a parameter estimation problem for 3D scene structure and motion. Our energy minimization formulation incorporates blur model constraints, moving object priors, and scene structure. Extensive evaluations on synthetic and real-world datasets demonstrate significant improvements in deblurring quality, scene flow accuracy, and segmentation performance over state-of-the-art methods.

Introduction: The Motion Blur Dilemma

In dynamic environments, such as those encountered by autonomous vehicles, stereo video often suffers from motion blur. This creates a "chicken-and-egg" problem for computer vision: accurate deblurring requires precise motion kernels, yet estimating motion (scene flow) typically requires sharp images to identify matching features. Furthermore, segmenting moving objects from the background is hindered when object boundaries are smeared by blur.

Traditionally, these tasks—deblurring, scene flow estimation, and object segmentation—have been addressed in isolation. This paper argues that such a disjointed approach fails to exploit the underlying 3D geometry that links them. By solving them jointly, we can use the estimated 3D structure to inform the deblurring process and vice versa.

Methodology: A Unified Framework

The core of the proposed method is a unified energy minimization framework. The scene is modeled using a piece-wise planar representation, where the environment is treated as a collection of small, rigidly moving flat tiles (superpixels). This model allows the complex dynamics of an urban scene to be expressed through a manageable set of 3D motion and structure parameters.

Energy Minimization

The energy function consists of three critical components:

  1. Data Term: Ensures consistency between deblurred images across stereo views and time, while requiring that the recovered sharp image, when re-blurred with the estimated motion kernel, matches the original input.
  2. Smoothness Term: Enforces spatial consistency, encouraging neighboring tiles to share similar 3D orientations and velocities, particularly within the same object boundaries.
  3. Regularization Term: Employs Total Variation (TV) to minimize noise and artifacts in the recovered latent images.

Semantic Priors and Iterative Optimization

To handle the complexity of real-world motion, the framework incorporates semantic segmentation (via ResNet38). By identifying categories like "road" or "car" initially, the system can apply specific motion priors—for instance, knowing that road pixels should follow camera ego-motion while car pixels may move independently.

Moving object segmentation identifying the cyclist and van.

The optimization is performed iteratively. The system first fixes the image to solve for motion and segmentation, then fixes the motion to solve for the sharp image. Usually, three iterations are sufficient to converge on a high-quality result.

Experimental Evaluation

The approach was tested on the KITTI dataset (using synthetic blurred versions BlurData-1 and BlurData-2) and real-world stereo sequences.

Quantitative Results

  • Deblurring: The method achieved a PSNR of approximately 30.50 dB on BlurData-1, a 1-2 dB improvement over previous stereo deblurring benchmarks.
  • Scene Flow: Optical flow error was reduced to 9.83%, compared to ~13.6% for non-joint methods like PRSF.
  • Segmentation: The moving object segmentation achieved an F-measure of 0.7426, significantly outperforming baseline methods that typically score between 0.60 and 0.67.
Visualization of the estimated 3D scene flow.

Qualitatively, the joint approach produces much sharper edges and fewer "ringing" artifacts. By incorporating segmentation into the loop, the algorithm handles motion boundaries more effectively, preventing the background from bleeding into the foreground objects during deblurring.

Conclusion

By treating deblurring, scene flow, and segmentation as a single, integrated problem, this framework overcomes the limitations of sequential processing. While the current implementation is computationally intensive (taking roughly 35 minutes per scene), it sets a new standard for accuracy in processing dynamic stereo video. Future work will focus on extending the temporal window beyond three frames and optimizing the system for near-real-time performance.

Summary updated

Abstract

Stereo videos for the dynamic scenes often show unpleasant blurred effects due to the camera motion and the multiple moving objects with large depth variations. Given consecutive blurred stereo video frames, we aim to recover the latent clean images, estimate the 3D scene flow and segment the multiple moving objects. These three tasks have been previously addressed separately, which fail to exploit the internal connections among these tasks and cannot achieve optimality. In this paper, we propose to jointly solve these three tasks in a unified framework by exploiting their intrinsic connections. To this end, we represent the dynamic scenes with the piece-wise planar model, which exploits the local structure of the scene and expresses various dynamic scenes. Under our model, these three tasks are naturally connected and expressed as the parameter estimation of 3D scene structure and camera motion (structure and motion for the dynamic scenes). By exploiting the blur model constraint, the moving objects and the 3D scene structure, we reach an energy minimization formulation for joint deblurring, scene flow and segmentation. We evaluate our approach extensively on both synthetic datasets and publicly available real datasets with fast-moving objects, camera motion, uncontrolled lighting conditions and shadows. Experimental results demonstrate that our method can achieve significant improvement in stereo video deblurring, scene flow estimation and moving object segmentation, over state-of-the-art methods.
BibTeX
@article{pan2020joint,
  author = {Liyuan Pan and Yuchao Dai and Miaomiao Liu and Fatih Porikli and Quan Pan},
  title = {{Joint Stereo Video Deblurring, Scene Flow Estimation and Moving Object Segmentation}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=QnfmqlDUXU},
}
Figure