Planar Object Tracking via Weighted Optical Flow

Jonáš Šerých, Jiří MatasWACV 2023

TL;DR

This paper introduces WOFT, a novel planar object tracking method that leverages dense optical flow and a new differentiable module to estimate homographies. By assigning weights to optical flow correspondences, WOFT achieves state-of-the-art performance on challenging benchmarks, outperforming existing methods and demonstrating robustness to illumination changes and motion blur.

What Problem Does This Paper Address?

Rigid planar object tracking, a crucial task in computer vision with applications ranging from film post-production to augmented reality, involves estimating the 8-degrees-of-freedom pose (homography) of a planar object in a video sequence. Traditional methods often rely on keypoint detection and matching followed by robust estimators like RANSAC, which can be computationally expensive and sensitive to outliers. More recent deep learning approaches have focused on directly regressing homography parameters, but these can struggle with occlusions and motion blur. The paper identifies a gap in robust, end-to-end trainable planar object trackers that can handle challenging scenarios like strong perspective distortion, specular highlights, and motion blur without relying on computationally intensive outlier rejection mechanisms.

Methods

The core innovation of WOFT lies in its Weighted Flow Homography (WFH) module. This module takes dense optical flow correspondences (obtained from a pre-trained optical flow network like RAFT) and assigns a weight to each correspondence. These weights are learned by a dedicated neural network that analyzes the optical flow's correlation cost-volume. Crucially, the WFH module then estimates the homography using a weighted least squares approach. This weighting mechanism allows the network to effectively down-weight or zero-out incorrect flow correspondences (outliers), making the homography estimation robust without the need for explicit outlier rejection algorithms like RANSAC. The entire WFH module is designed to be differentiable, enabling end-to-end training of both the weight estimation network and the optical flow network.

The WOFT tracker itself incorporates several key components:

  • Pre-warping: To handle large pose differences between frames, the current frame is pre-warped using the homography from the last reliable frame. This reduces the optical flow estimation problem to a more manageable, smaller motion.
  • Global Optical Flow: Dense optical flow is computed between the template frame and the pre-warped current frame.
  • State Logic: A mechanism to detect if tracking has been "lost" based on the support set size of the estimated homography (i.e., the number of reliable correspondences). If lost, the tracker switches to a local optical flow estimation to attempt recovery.
  • Homography Estimation: The WFH module is used for estimating the homography.

Datasets

The paper evaluates WOFT on two standard planar object tracking datasets:

  • POT-210: A benchmark with 210 videos featuring 30 objects, covering challenges like scale change, in-plane rotation, perspective distortion, motion blur, occlusion, out-of-view, and unconstrained scenarios.
  • POIC: A dataset focusing on planar objects with significant illumination changes, including specular highlights.

The authors also mention a re-annotation of a subset of the POT-210 dataset due to observed inaccuracies in the original ground truth, which they found to account for a significant portion of their tracker's errors.

Key Insights and Ideas

  • Differentiable Robustness: The most significant insight is the development of a differentiable homography estimation module that achieves robustness to outliers by learning to weight correspondences. This bypasses the need for non-differentiable, iterative methods like RANSAC, enabling end-to-end training.
  • Optical Flow as a Rich Signal: The paper demonstrates that dense optical flow, when combined with learned weights derived from its cost-volume, provides a powerful signal for robust homography estimation, especially for low-textured objects where sparse keypoint methods might fail.
  • Pre-warping for Stability: The pre-warping step is crucial for handling large pose changes, effectively reducing the complexity of the optical flow problem and preventing error accumulation.
  • Ground Truth Matters: The analysis of the POT-210 dataset highlights the critical impact of ground truth quality on evaluating tracking performance, with re-annotation revealing significant inaccuracies in the original annotations.

Key Results

  • State-of-the-Art Performance: WOFT achieves state-of-the-art results on the POT-210, POT-280, and POIC datasets.
  • POT-210 Improvement: On POT-210, WOFT's error is half that of the best competing method, and it significantly outperforms other trackers on various challenge types.
  • Robustness: WOFT demonstrates robustness to illumination changes and motion blur, particularly on the POIC dataset.
  • Efficiency: While the RAFT optical flow computation is a bottleneck (275ms), the WFH module itself is very fast (a few milliseconds). A downscaled variant (WOFT↓3) achieves near real-time performance (19.2 FPS) while maintaining state-of-the-art accuracy.
  • Ablation Study: The ablation study confirms the importance of global flow, pre-warping, and weighted least squares estimation for achieving high performance.

Limitations and Future Work

  • Optical Flow Bottleneck: The primary limitation is the computational cost of the optical flow estimation, which is currently the dominant factor in WOFT's runtime. The authors acknowledge that optical flow is an active research area and anticipate faster, more accurate methods to emerge.
  • Scale Subset Issues: The tracker can struggle with significant scale changes when the pre-warp source frame is reset to identity, as the flow network might not handle such large transformations effectively.
  • Future Work: The authors suggest that the core idea of WFH—using learned flow weights from cost-volumes with differentiable weighted LSq—is applicable to other optical flow methods. They also plan to release their code, trained model, and the improved GT annotation of the POT-210 subset.

Summary updated

BibTeX
@inproceedings{serych2023planar,
  author = {Jonáš Šerých and Jiří Matas},
  title = {{Planar Object Tracking via Weighted Optical Flow}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure