Motion Prediction in Visual Object Tracking

Jianren Wang, Yihui He

In the field of computer vision, Visual Object Tracking (VOT) has recently been dominated by the "tracking-by-detection" paradigm. Most modern trackers focus heavily on building more powerful deep-learning feature extractors to recognize what an object looks like, often ignoring the physics of how it moves. In fact, many state-of-the-art trackers simply assume a "zero-velocity" model—essentially guessing the object hasn't moved much since the last frame.

In their paper, "Motion Prediction in Visual Object Tracking," Jianren Wang and Yihui He from Carnegie Mellon University argue that we’ve neglected a fundamental truth: objects follow predictable paths. By reintroducing robust motion prediction, they’ve set new state-of-the-art records on major benchmarks.

The Problem: Appearance is Not Enough

Modern tracking datasets are challenging, featuring random camera shakes, fast-moving targets, and "distractors"—objects that look almost identical to the target. Current deep-learning trackers (like Siamese networks) are excellent at appearance matching but often fail when the camera moves abruptly, when occlusions occur, or when distractors appear. If two similar cars pass each other, an appearance-only tracker might accidentally switch to the wrong one because it lacks a sense of trajectory.

Methodology: Decoupling and Estimation

The authors propose a multi-stage pipeline that can be wrapped around any "off-the-shelf" object detector, such as SiamRPN++ or SiamMask. The core innovation lies in how motion is handled:

  1. Camera Motion Decoupling: Before predicting where an object went, the system must account for where the camera moved. They use homography estimation to map the current frame back to a reference frame, effectively "stabilizing" the video for the motion model.
  2. Kalman Filter Prediction: Once camera motion is removed, the object's trajectory becomes much smoother. A standard Kalman filter with a constant velocity model predicts the object's next state (position, size, and velocity).
Figure 1: Visualization of the motion prediction vector (arrow) and state estimation (red overlay) for a fast-moving athlete.
  1. Adaptive Search Region: Instead of a fixed search window, the system dynamically resizes the search area based on the object's predicted velocity and size. If an object is moving fast, the search area grows; if it's slow or small, it shrinks to filter out background noise.

Key Insights: Motion as a Tie-Breaker

A critical finding of this research is that decoupling is essential. Predicting motion in raw camera coordinates is noisy due to handheld shakes. By decoupling camera motion, the object's path becomes a predictable physical trajectory. Furthermore, when appearance is ambiguous (e.g., two identical runners in a race), the motion model acts as a prior that prevents the tracker from jumping to a distractor that is physically impossible to reach in a single frame.

Figure 5: The motion-aware framework helps maintain track on specific targets even in crowded environments with potential distractors.

Results and Performance

The results established new state-of-the-art (SOTA) performance across multiple benchmarks:

  • VOT-2016: Improved the Expected Average Overlap (EAO) from 0.472 to 0.505.
  • VOT-2018: Improved EAO from 0.410 to 0.431.
  • Robustness: On VOT-2018, their method achieved the lowest "Robustness" score (indicating fewer tracking failures) among all Siamese-based trackers, dropping from 0.241 to 0.203.

Despite the added complexity of motion prediction, the modules are lightweight. The system still runs at roughly 30 FPS on a modern GPU (RTX 2080Ti), making it suitable for real-time applications.

Conclusion

This work demonstrates that combining "old school" estimation techniques like Kalman Filters and Homography with "new school" deep learning detection yields better results than either could achieve alone. While challenges remain for highly non-rigid objects, this framework provides a robust foundation for more physics-aware visual tracking.

Summary updated

BibTeX
@inproceedings{wang2020motion,
  author = {Jianren Wang and Yihui He},
  title = {{Motion Prediction in Visual Object Tracking}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  doi = {10.1109/iros45743.2020.9341158},
}
Figure