Aggregation of local parametric candidates with exemplar-based occlusion handling for optical flow

Denis Fortun, Patrick Bouthemy, Charles Kervrann

TL;DR

This paper introduces AggregFlow, a novel two-step approach for computing optical flow that excels at handling large displacements and occlusions. By first generating a rich set of local motion candidates and then aggregating them through a discrete optimization process that jointly estimates motion and occlusion, AggregFlow achieves state-of-the-art results on challenging benchmarks.

What Problem Does This Paper Address?

Estimating optical flow, which describes the motion of pixels between consecutive video frames, is a fundamental problem in computer vision with applications ranging from object tracking to autonomous navigation. A significant challenge in optical flow estimation is accurately handling scenarios with large object displacements and occlusions (regions where objects disappear from view). Traditional methods often struggle in these situations, leading to inaccurate flow fields. This paper aims to address this by proposing a robust method that explicitly accounts for both large motions and occluded regions.

Methods

AggregFlow employs a two-step paradigm:

  1. Local Motion Candidate Generation: This initial step focuses on generating a diverse set of potential motion vectors for each pixel. It combines two key techniques:
  2. Local Parametric Motion Estimation: Affine motion models are estimated within overlapping square patches of varying sizes. This allows for capturing local motion variations and avoids the need for explicit image segmentation.
  3. Patch Correspondences: To handle large displacements, the method leverages patch matching to find corresponding patches between frames, providing initial motion estimates.
  4. Exemplar-based Occlusion Filling: For occluded regions, motion candidates are "copied" from similar non-occluded regions using an exemplar-based approach. Additionally, dominant camera motion is estimated and incorporated as a candidate.
  5. Occlusion Confidence Map: A coarse occlusion confidence map is generated based on patch-level consistency checks, which will guide the subsequent occlusion estimation.
  6. Global Aggregation: In the second step, a global energy function is minimized to select the best motion vector for each pixel and simultaneously estimate a binary occlusion map. This is formulated as a discrete optimization problem, leveraging the candidates generated in the first step. The energy function includes:
  7. Data Term: Enforces image intensity and gradient constancy for visible pixels and uses an exemplar-based term for occluded pixels.
  8. Occlusion Term: Penalizes occlusion occurrence, guided by the occlusion confidence map from the first step.
  9. Regularization Terms: Promote piecewise smoothness for both the motion field and the occlusion map.

The optimization is performed iteratively, alternating between estimating the motion field and the occlusion map.

Datasets

The authors evaluate AggregFlow on two prominent optical flow benchmarks:

  • MPI Sintel Flow Dataset: This dataset is characterized by long-range motion, motion blur, non-rigid motion, and wide occluded areas, making it particularly challenging.
  • Middlebury Flow Dataset: This benchmark features sequences with smaller displacements, focusing on recovering complex smooth deformations, motion discontinuities, and motion details.

The paper does not introduce new datasets.

Key Insights and Ideas

  • Patch-based Affine Models are Powerful: The paper demonstrates that combining affine motion models estimated on overlapping patches of various sizes, along with patch correspondences, can generate highly accurate motion candidates, even for large displacements.
  • Joint Motion and Occlusion Estimation is Crucial: AggregFlow's success hinges on its ability to estimate motion and occlusion simultaneously. This prevents the "chicken-and-egg" problem where occlusion detection relies on motion, and motion estimation is hindered by unknown occlusions.
  • Exemplar-based Occlusion Filling Works: The exemplar-based approach for filling occluded regions with motion vectors proves effective, especially for large occlusions, outperforming traditional diffusion-based methods.
  • Occlusion Confidence Map Guides Sparsity: The occlusion confidence map, derived from the first step, acts as a crucial prior, guiding the sparsity constraint for occlusion detection and preventing it from being overly coupled with the motion estimation.

Key Results

  • State-of-the-Art Performance: AggregFlow achieves state-of-the-art results on the MPI Sintel benchmark, particularly excelling in the "unmatched" category (occluded regions) and for large displacements.
  • Significant Improvement with Occlusion Handling: The full AggregFlow method (with occlusion handling) shows a substantial improvement in average Endpoint Error (EPE) compared to a version without occlusion terms (e.g., 6.002 vs. 8.417 on MPI Sintel training sequences).
  • Competitive on Middlebury: While the Middlebury benchmark has smaller displacements, AggregFlow remains competitive, demonstrating its ability to recover motion details and discontinuities.
  • Visual Evidence: Figures in the paper visually demonstrate the superior performance of AggregFlow in occluded regions and its ability to preserve motion discontinuities compared to other methods.

Limitations and Future Work

  • Computational Cost: The patch correspondence sub-step, especially for large patch sizes, can be computationally intensive. The authors suggest GPU implementation for significant speedups.
  • Matching Errors: Future work could explore more robust and discriminative distance metrics for patch matching to further reduce matching errors.
  • Foggy Conditions: The method's performance can be affected by atmospheric effects like fog, where global variational approaches might be more suitable. Incorporating more sophisticated feature correspondence techniques could mitigate this.
  • GPU Implementation: The authors explicitly mention that a GPU implementation would greatly improve computation efficiency.

Summary updated

Abstract

Handling all together large displacements, motion details and occlusions remains an open issue for reliable computation of optical flow in a video sequence. We propose a two-step aggregation paradigm to address this problem. The idea is to supply local motion candidates at every pixel in a first step, and then to combine them to determine the global optical flow field in a second step. We exploit local parametric estimations combined with patch correspondences and we experimentally demonstrate that they are sufficient to produce highly accurate motion candidates. The aggregation step is designed as the discrete optimization of a global regularized energy. The occlusion map is estimated jointly with the flow field throughout the two steps. We propose a generic exemplar-based approach for occlusion filling with motion vectors. We achieve state-of-the-art results in computer vision benchmarks, with particularly significant improvements in the case of large displacements and occlusions.
BibTeX
@article{fortun2016aggregation,
  author = {Denis Fortun and Patrick Bouthemy and Charles Kervrann},
  title = {{Aggregation of local parametric candidates with exemplar-based occlusion handling for optical flow}},
  journal = {Comput. Vis. Image Underst.},
  year = {2016},
  doi = {10.1016/j.cviu.2015.11.020},
}
Figure