DeepFlow: Large Displacement Optical Flow with Deep Matching

Philippe Weinzaepfel, Jérôme Revaud, Zaïd Harchaoui, Cordelia Schmid

Abstract

Optical flow computation is a key component in many computer vision systems designed for tasks such as action detection or activity recognition. However, despite several major advances over the last decade, handling large displacement in optical flow remains an open problem. Inspired by the large displacement optical flow of Brox and Malik, our approach, termed Deep Flow, blends a matching algorithm with a variational approach for optical flow. We propose a descriptor matching algorithm, tailored to the optical flow problem, that allows to boost performance on fast motions. The matching algorithm builds upon a multi-stage architecture with 6 layers, interleaving convolutions and max-pooling, a construction akin to deep convolutional nets. Using dense sampling, it allows to efficiently retrieve quasi-dense correspondences, and enjoys a built-in smoothing effect on descriptors matches, a valuable asset for integration into an energy minimization framework for optical flow estimation. Deep Flow efficiently handles large displacements occurring in realistic videos, and shows competitive performance on optical flow benchmarks. Furthermore, it sets a new state-of-the-art on the MPI-Sintel dataset.

Introduction

Optical flow—the estimation of per-pixel motion between consecutive video frames—is fundamental to computer vision. While traditional variational methods excel at capturing small, smooth motions using local gradients, they frequently fail when objects move rapidly across the screen. This "large displacement" problem occurs because the search space for local optimization becomes too vast.

Previous attempts to solve this involved incorporating sparse descriptor matches (like SIFT) as "hints." However, these descriptors are often rigid and sparse, making them brittle when objects rotate or deform. DeepFlow addresses this by integrating a novel Deep Matching algorithm into a refined variational framework, producing dense, deformable correspondences that significantly improve performance on fast-moving sequences.

Deep Matching: A Hierarchical Approach to Correspondences

The core innovation of this work is the Deep Matching algorithm. Inspired by the architecture of Deep Convolutional Neural Networks (ConvNets), it utilizes a multi-stage process to find correspondences.

Bottom-Up Aggregation and Deformability

The process begins by matching small $4 \times 4$ pixel patches. Unlike traditional rigid descriptors, Deep Matching allows for non-rigid motion by splitting patches into quadrants that can move independently within a restricted range. This flexibility allows the algorithm to account for scaling and rotation, which are common in realistic video sequences.

The Deep Matching algorithm allows for deformable quadrants (b'), enabling it to handle non-rigid motion better than rigid grids (a).

As the algorithm progresses through its 6 layers, it uses max-pooling and sub-sampling to aggregate these quadrant matches into larger, more discriminative "virtual" patches (e.g., $8 \times 8, 16 \times 16$). This creates a multi-size response pyramid that is robust to noise and repetitive textures, eventually producing "quasi-dense" correspondences covering roughly 80% of the image.

The DeepFlow Variational Framework

The correspondences generated by Deep Matching are integrated into a variational energy minimization framework. The authors introduce three key refinements to the standard model to better leverage these matches:

  1. Matching Term: A specific energy term that encourages the flow to align with the Deep Matching correspondences.
  2. Data Term Normalization: Normalizing the data term prevents areas with high image gradients from disproportionately influencing the optimization.
  3. Scale-Dependent Weighting: Weights for the matching term are adjusted across the image pyramid, ensuring that matching "hints" guide the coarse scales without overpowering fine-grained details at the final resolution.
Overview of the DeepFlow pipeline, illustrating the transition from patch matching and aggregation to the final optical flow estimation.

Experimental Evaluation

DeepFlow was evaluated on three major benchmarks: MPI-Sintel, KITTI, and Middlebury.

  • MPI-Sintel: On this challenging dataset featuring large motions and motion blur, DeepFlow set a new state-of-the-art. Specifically, for pixels moving faster than 40 pixels per frame, it reduced the Average Endpoint Error (EPE) from 57.3 to 44.1 compared to previous leading methods.
  • KITTI & Middlebury: The method proved robust across real-world driving sequences and traditional small-displacement benchmarks, showing that the added capability for large displacements does not degrade performance on simpler cases.

Conclusion

DeepFlow demonstrates that treating the matching problem as a deep, hierarchical process provides a built-in smoothing effect and multi-scale robustness essential for optical flow. By prioritizing quasi-dense correspondences and non-rigidity, the framework effectively bridges the gap between local variational optimization and global descriptor matching. While the current implementation takes approximately 20 seconds per frame on a CPU, it establishes a powerful foundation for future real-time, learning-based motion estimation systems.

Summary updated

Abstract

Optical flow computation is a key component in many computer vision systems designed for tasks such as action detection or activity recognition. However, despite several major advances over the last decade, handling large displacement in optical flow remains an open problem. Inspired by the large displacement optical flow of Brox and Malik, our approach, termed Deep Flow, blends a matching algorithm with a variational approach for optical flow. We propose a descriptor matching algorithm, tailored to the optical flow problem, that allows to boost performance on fast motions. The matching algorithm builds upon a multi-stage architecture with 6 layers, interleaving convolutions and max-pooling, a construction akin to deep convolutional nets. Using dense sampling, it allows to efficiently retrieve quasi-dense correspondences, and enjoys a built-in smoothing effect on descriptors matches, a valuable asset for integration into an energy minimization framework for optical flow estimation. Deep Flow efficiently handles large displacements occurring in realistic videos, and shows competitive performance on optical flow benchmarks. Furthermore, it sets a new state-of-the-art on the MPI-Sintel dataset.
BibTeX
@inproceedings{weinzaepfel2013deepflow,
  author = {Philippe Weinzaepfel and Jérôme Revaud and Zaïd Harchaoui and Cordelia Schmid},
  title = {{DeepFlow: Large Displacement Optical Flow with Deep Matching}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2013},
  url = {https://openreview.net/forum?id=By-Iv-fO-S},
}
Figure