Abstract
This study investigates the role of motion information in deep visual tracking by examining whether neural networks implicitly learn optical flow or benefit from its explicit inclusion. Using the GOTURN tracker as a baseline, researchers demonstrate that while standard trackers develop effective implicit motion representations, providing explicit optical flow enables the use of significantly more efficient, "lighter" architectures without compromising tracking accuracy.
Introduction: The Role of Motion in Tracking
Single-target visual tracking—the task of following a generic object through a video sequence after a single initial observation—is a fundamental challenge in computer vision. While modern trackers predominantly rely on deep neural networks to learn discriminative features, these models often operate as "black boxes."
Intuitively, motion (the displacement of pixels between frames) is a critical cue for tracking. However, most deep trackers are trained on raw RGB image frames. This raises two pivotal questions: Do these networks implicitly learn to calculate optical flow? And can explicit optical flow, provided by traditional algorithms, improve network efficiency or robustness?
Methodology
The researchers employed the GOTURN (Generic Object Tracking Using Regression Networks) tracker as their baseline. GOTURN typically processes two inputs: a crop of the previous frame and a crop of the current frame. To isolate the impact of motion information, the authors developed three architectural variants:
- Flow1s: Utilizes only the optical flow between the two frames.
- Flow2s: Utilizes the optical flow and the current frame (replacing the previous frame).
- Flow3s: Utilizes the optical flow, the current frame, and the previous frame.
Explicit optical flow was calculated using the Farneback method, a traditional non-deep algorithm. Additionally, the researchers tested "light" versions of these architectures by reducing parameters in the fully-connected layers to evaluate how explicit flow supports model compression.
Datasets and Evaluation
Models were trained on the ILSVRC (ImageNet Video) dataset, which provides thousands of videos with annotated bounding boxes. Evaluation was conducted using the VOT2015 and VOT2016 benchmarks, focusing on standard metrics: accuracy (Overlap) and robustness (Failures).
Key Insights
Implicit Learning and Redundancy
The performance of the Flow2s model (Flow + Current Frame) was nearly identical to the baseline GOTURN (Previous + Current Frame). This suggests that GOTURN implicitly calculates a representation similar to optical flow from raw RGB frames. Furthermore, adding explicit flow to the full baseline (Flow3s) yielded no significant performance boost, indicating that the network does not gain novel information but rather finds alternative ways to represent the same motion.
The Necessity of Appearance
The Flow1s model, which relied solely on motion, performed poorly. This confirms that motion alone is insufficient for robust tracking; the network requires "static" appearance features from RGB frames to effectively distinguish the target from its background.
Efficiency and Resilience
The most significant finding relates to model efficiency. When the architectures were "downsized" to create lightweight versions, the performance of the standard RGB-only GOTURN dropped by 28%. In contrast, the flow-augmented Flow2s model experienced only a 13% performance decrease. This suggests that explicit flow simplifies the learning task, allowing smaller networks to achieve performance levels previously reserved for much larger models.
Results
On the VOT2015 benchmark, the baseline GOTURN achieved an Expected Overlap of 0.1691. The Flow2s variant achieved a comparable 0.1557. Notably, Flow2s demonstrated a slightly lower failure rate (2.47) than the baseline (2.69) in the full-sized architecture.
In lightweight tests, Flow2s consistently outperformed the light version of GOTURN in both overlap and failure rates, highlighting its utility for resource-constrained applications.
Conclusion and Future Work
While this study uses the GOTURN architecture, the findings suggest broad implications for deep tracking. Explicit flow can act as a powerful prior that enables model compression. Future work could investigate whether these results hold for more modern architectures, such as Siamese-RPN or Transformer-based trackers, and whether deep-learning-based flow estimators could further enhance performance in challenging scenarios like extreme motion blur or occlusion.
Summary updated