Improved Discrete Optical Flow Estimation With Triple Image Matching Cost

Fei Yang, Yongmei Cheng, Joost Van De Weijer, Mikhail G. MozerovIEEE Access 2020 · OpenReview

Abstract

Optical flow estimation is a fundamental task in computer vision, yet it remains highly sensitive to occlusions where pixels disappear or reappear between frames. Traditional methods typically rely on two consecutive frames, often leading to significant errors in these regions. This paper introduces TIMCflow, a novel approach that utilizes three consecutive video frames to calculate a "Triple Image Matching Cost." By exploiting the temporal symmetry of motion—where a pixel occluded in the future is frequently visible in the past—TIMCflow achieves up to a 50% reduction in error within occluded regions on the MPI-Sintel dataset, while also improving robustness in non-occluded areas.

Introduction: The Challenge of Occlusions

Optical flow—the pattern of apparent motion of objects in a visual scene—is critical for applications ranging from autonomous driving to 3D reconstruction. A major hurdle for current algorithms is the presence of occlusions and dis-occlusions. When an object moves, it inevitably covers some background pixels and uncovers others. In a standard two-frame setup, a pixel that is visible in frame $t$ but disappears in frame $t+1$ has no valid match, forcing the algorithm to rely on post-processing or interpolation to guess the motion.

While many researchers have attempted to use multi-frame information for post-processing flow correction or pre-processing prediction, this paper argues for a more fundamental shift: integrating multi-frame data directly into the likelihood function (matching cost volume) calculation.

Methods: Leveraging Temporal Symmetry

The core innovation of TIMCflow is the Triple Image Matching Cost (TIMC). Instead of the standard forward-only comparison, the algorithm evaluates matches across three frames: $f^{t-1}, f^t,$ and $f^{t+1}$.

The Symmetry Principle

The method relies on the assumption that motion is locally constant over short time intervals. Under this assumption, the forward motion vector ($v^{fr}$) from frame $t$ to $t+1$ should be approximately the negative of the backward motion vector ($v^{bk}$) from frame $t$ to $t-1$.

Conceptual overview of the triple image matching cost and the symmetry principle.

The "Min" Trick in Cost Calculation

To handle occlusions effectively, the authors propose a composite cost function: $$C = \lambda_1 C^{fr} + \lambda_2 C^{bk} + \lambda_3 \min(C^{fr}, C^{bk})$$

The inclusion of the $\min(C^{fr}, C^{bk})$ term is crucial. If a pixel is occluded in the future frame ($t+1$), the forward cost ($C^{fr}$) will be high. However, if that same pixel was visible in the previous frame ($t-1$), the backward cost ($C^{bk}$) will be low. By taking the minimum, the algorithm automatically favors the frame where the pixel is visible, providing a reliable match even when a two-frame method would fail.

Results and Performance

The researchers evaluated TIMCflow on several benchmarks, most notably the MPI-Sintel dataset, which is known for its complex motion and heavy occlusions.

  • Occlusion Mastery: The method demonstrated a remarkable improvement, reducing flow estimation errors in occluded regions by up to 50% compared to the baseline DCFlow architecture.
  • General Robustness: Even in non-occluded regions, the triple-matching approach improved accuracy by approximately 9%. The additional data from the third frame creates a more stable likelihood function, making the algorithm more resilient to noise and lighting changes.
  • Efficiency: Despite the added complexity of a third frame, the authors optimized the pipeline such that their implementation was actually faster than the standard two-frame baseline in certain configurations (3.06s vs 4.24s).

Conclusion

TIMCflow demonstrates that treating occlusion as a "temporally shifted visibility" problem rather than a simple noise problem yields significant dividends. By moving multi-frame integration from post-processing to the initial matching cost calculation, the method achieves state-of-the-art results on challenging benchmarks. Future work aims to refine outlier handling to preserve the high-quality flow values generated by this triple-frame approach, particularly in high-acceleration scenarios where the constant motion assumption may be challenged.

Summary updated

Abstract

Approaches that use more than two consecutive video frames in the optical flow estimation have a long research history. However, almost all such methods utilize extra information for a pre-processing flow prediction or for a post-processing flow correction and filtering. In contrast, this paper differs from previously developed techniques. We propose a new algorithm for the likelihood function calculation (alternatively the matching cost volume) that is used in the maximum a posteriori estimation. We exploit the fact that in general, optical flow is locally constant in the sense of time and the likelihood function depends on both the previous and the future frame. Implementation of our idea increases the robustness of optical flow estimation. As a result, our method outperforms 9% over the DCFlow technique, which we use as prototype for our CNN based computation architecture, on the most challenging MPI-Sintel dataset for the non-occluded mask metric. Furthermore, our approach considerably increases the accuracy of the flow estimation for the matching cost processing, consequently outperforming the original DCFlow algorithm results up to 50% in occluded regions and up to 9% in non-occluded regions on the MPI-Sintel dataset. The experimental section shows that the proposed method achieves state-of-the-arts results especially on the MPI-Sintel dataset.
BibTeX
@inproceedings{yang2020improved,
  author = {Fei Yang and Yongmei Cheng and Joost Van De Weijer and Mikhail G. Mozerov},
  title = {{Improved Discrete Optical Flow Estimation With Triple Image Matching Cost}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=FR7EhCIQSN},
}
Figure