Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation

Christian Bailer, Bertram Taetz, Didier StrickerIEEE TPAMI 2019 · OpenReview

TL;DR

This paper introduces "Flow Fields," a novel dense correspondence field approach for optical flow estimation that significantly reduces outliers compared to traditional methods. By leveraging a multi-scale patch matching strategy without explicit regularization, Flow Fields achieve state-of-the-art accuracy on challenging datasets, outperforming existing methods and even improving upon the popular EpicFlow algorithm.

What Problem Does This Paper Address?

Estimating optical flow, the motion of pixels between consecutive frames in a video, is crucial for many computer vision tasks. A major challenge, especially for large displacements, is dealing with "outliers" – incorrect correspondences that can derail subsequent processing. Existing methods often rely on either sparse descriptors (which leave gaps) or dense approximate nearest neighbor fields (ANNFs), which are prone to outliers because they prioritize visual similarity over actual motion. This paper tackles the problem of generating dense, accurate, and outlier-resistant correspondence fields that are well-suited for optical flow estimation, particularly for scenarios with large pixel movements.

Methods

The core of the proposed "Flow Fields" approach lies in a novel multi-scale patch matching strategy. Unlike ANNF methods that find the visually most similar pixel, Flow Fields aim to find the pixel that best represents the optical flow. Key aspects include:

  • Dense Correspondence Fields: The output is a dense field, meaning every pixel has a corresponding motion vector.
  • Multi-Scale Matching: The algorithm iteratively refines the flow estimation across different image scales. This "outlier sieve" effect allows outliers that are problematic at one scale to be filtered out at another.
  • Patch Matching with Localized Search: Instead of a global search, the algorithm uses patch matching but restricts the search in "flow space" to a small neighborhood. This, combined with multi-scale propagation, compensates for the limited local search and prevents the introduction of new, resistant outliers.
  • No Explicit Regularization: The method avoids traditional regularization techniques (like median filtering or explicit smoothness priors) that can sometimes smooth out important motion discontinuities or introduce their own issues. The outlier resistance is achieved through the data-driven multi-scale matching process itself.
  • Enhanced Outlier Filtering: The paper also proposes improvements to outlier filtering, including a novel two-way consistency check and region-based filtering.

The authors also introduce variations: "Flow Fields+" for improved accuracy and "Flow Fields+ Fast" for increased speed, including a "Flow Fields+ Fast x2" variant that omits the finest scale for further acceleration.

Datasets

The authors evaluate their approach on several standard optical flow datasets:

  • MPI-Sintel: Known for its animated sequences with large motions and challenging visual effects (clean and final versions).
  • KITTI 2012 & 2015: Datasets captured from a driving car, featuring realistic urban scenes and significant motion.
  • Middlebury: A dataset traditionally used for optical flow estimation with relatively small displacements.

Key Insights and Ideas

  • Outlier Sieve Effect: The multi-scale nature of the algorithm acts as an "outlier sieve." Outliers that might be accepted at a coarse scale can be rejected at finer scales, and vice-versa, leading to a more robust final flow field.
  • Locality in Flow Space, Non-Locality in Image Space: The approach balances local constraints in the flow (movement) with the ability to propagate matches over large distances in the image space. This is crucial for handling large displacements.
  • Data-Driven Outlier Rejection: The core idea is that outlier resistance can be achieved through clever data-driven matching strategies rather than relying on explicit geometric or smoothness priors, which can be problematic for optical flow.
  • Multi-Scale Benefits for Small Objects: The multi-scale approach is shown to be particularly effective at preserving small objects and details that might be lost with single-scale methods or traditional regularization.

Key Results

  • Superior to ANNF: Flow Fields significantly outperform ANNF methods in terms of outlier reduction and optical flow accuracy.
  • Outperforms EpicFlow: When used to initialize the state-of-the-art EpicFlow algorithm, Flow Fields achieve substantially better results than EpicFlow initialized with its original descriptor matching technique.
  • State-of-the-Art Performance: The proposed methods achieve competitive or state-of-the-art results on MPI-Sintel, KITTI 2012, KITTI 2015, and Middlebury datasets. For instance, on the MPI-Sintel clean set, their Flow Fields+ approach achieved an EPE of 0.820, reportedly the best submission at the time of writing.
  • Speed Improvements: The "Flow Fields+ Fast" and "Flow Fields+ Fast x2" variants offer significant speedups (e.g., 1.2s runtime for Fast x2 on MPI-Sintel) while maintaining competitive accuracy, often outperforming the original conference version.
  • Robustness: The method demonstrates robustness to noise and blur, and effectively handles repetitive patterns and textureless regions.

Limitations and Future Work

  • Outlier Filtering of Important Details: While generally good, the outlier filtering can sometimes remove important details, especially when they are associated with very fast-moving objects or subtle motion discontinuities (as shown in Figure 16 b).
  • Occluded Regions: Like most data-based methods, Flow Fields still struggle with occluded regions, where they rely on interpolation (like EpicFlow) for completion.
  • Future Work: The authors suggest that better interpolation techniques for occluded areas could further improve results. They also mention the development of a CNN-based data term for their approach, indicating a direction towards integrating learned features. The paper also hints at further experiments and insights in the extended version.

Summary updated

Abstract

Modern large displacement optical flow algorithms usually use an initialization by either sparse descriptor matching techniques or dense approximate nearest neighbor fields. While the latter have the advantage of being dense, they have the major disadvantage of being very outlier-prone as they are not designed to find the optical flow, but the visually most similar correspondence. In this article we present a dense correspondence field approach that is much less outlier-prone and thus much better suited for optical flow estimation than approximate nearest neighbor fields. Our approach does not require explicit regularization, smoothing (like median filtering) or a new data term. Instead we solely rely on patch matching techniques and a novel multi-scale matching strategy. We also present enhancements for outlier filtering. We show that our approach is better suited for large displacement optical flow estimation than modern descriptor matching techniques. We do so by initializing EpicFlow with our approach instead of their originally used stateof-the-art descriptor matching technique. We significantly outperform the original EpicFlow on MPI-Sintel, KITTI 2012, KITTI 2015 and Middlebury. In this extended article of our former conference publication we further improve our approach in matching accuracy as well as runtime and present more experiments and insights.
BibTeX
@article{bailer2019flow,
  author = {Christian Bailer and Bertram Taetz and Didier Stricker},
  title = {{Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2019},
  url = {https://openreview.net/forum?id=M8xYvxFc5qaK},
}
Figure