Abstract
Stereo videos of dynamic scenes often exhibit unpleasant blur effects due to camera motion and multiple moving objects with significant depth variations. This paper proposes a unified framework to jointly recover latent clean images, estimate 3D scene flow, and segment multiple moving objects from consecutive blurred stereo frames. By representing dynamic scenes with a piece-wise planar model, we exploit the intrinsic connections between these tasks, expressing them as a parameter estimation problem for 3D scene structure and motion. Our energy minimization formulation incorporates blur model constraints, moving object priors, and scene structure. Extensive evaluations on synthetic and real-world datasets demonstrate significant improvements in deblurring quality, scene flow accuracy, and segmentation performance over state-of-the-art methods.
Introduction: The Motion Blur Dilemma
In dynamic environments, such as those encountered by autonomous vehicles, stereo video often suffers from motion blur. This creates a "chicken-and-egg" problem for computer vision: accurate deblurring requires precise motion kernels, yet estimating motion (scene flow) typically requires sharp images to identify matching features. Furthermore, segmenting moving objects from the background is hindered when object boundaries are smeared by blur.
Traditionally, these tasks—deblurring, scene flow estimation, and object segmentation—have been addressed in isolation. This paper argues that such a disjointed approach fails to exploit the underlying 3D geometry that links them. By solving them jointly, we can use the estimated 3D structure to inform the deblurring process and vice versa.
Methodology: A Unified Framework
The core of the proposed method is a unified energy minimization framework. The scene is modeled using a piece-wise planar representation, where the environment is treated as a collection of small, rigidly moving flat tiles (superpixels). This model allows the complex dynamics of an urban scene to be expressed through a manageable set of 3D motion and structure parameters.
Energy Minimization
The energy function consists of three critical components:
- Data Term: Ensures consistency between deblurred images across stereo views and time, while requiring that the recovered sharp image, when re-blurred with the estimated motion kernel, matches the original input.
- Smoothness Term: Enforces spatial consistency, encouraging neighboring tiles to share similar 3D orientations and velocities, particularly within the same object boundaries.
- Regularization Term: Employs Total Variation (TV) to minimize noise and artifacts in the recovered latent images.
Semantic Priors and Iterative Optimization
To handle the complexity of real-world motion, the framework incorporates semantic segmentation (via ResNet38). By identifying categories like "road" or "car" initially, the system can apply specific motion priors—for instance, knowing that road pixels should follow camera ego-motion while car pixels may move independently.
The optimization is performed iteratively. The system first fixes the image to solve for motion and segmentation, then fixes the motion to solve for the sharp image. Usually, three iterations are sufficient to converge on a high-quality result.
Experimental Evaluation
The approach was tested on the KITTI dataset (using synthetic blurred versions BlurData-1 and BlurData-2) and real-world stereo sequences.
Quantitative Results
- Deblurring: The method achieved a PSNR of approximately 30.50 dB on BlurData-1, a 1-2 dB improvement over previous stereo deblurring benchmarks.
- Scene Flow: Optical flow error was reduced to 9.83%, compared to ~13.6% for non-joint methods like PRSF.
- Segmentation: The moving object segmentation achieved an F-measure of 0.7426, significantly outperforming baseline methods that typically score between 0.60 and 0.67.
Qualitatively, the joint approach produces much sharper edges and fewer "ringing" artifacts. By incorporating segmentation into the loop, the algorithm handles motion boundaries more effectively, preventing the background from bleeding into the foreground objects during deblurring.
Conclusion
By treating deblurring, scene flow, and segmentation as a single, integrated problem, this framework overcomes the limitations of sequential processing. While the current implementation is computationally intensive (taking roughly 35 minutes per scene), it sets a new standard for accuracy in processing dynamic stereo video. Future work will focus on extending the temporal window beyond three frames and optimizing the system for near-real-time performance.
Summary updated