Deep Flow Collaborative Network for Online Visual Tracking

Peidong Liu, Xiyu Yan, Yong Jiang, Shu-Tao Xia

Abstract: Visual tracking using deep Convolutional Neural Networks (CNNs) has achieved state-of-the-art accuracy but suffers from high computational costs. This paper introduces the Deep Flow Collaborative Network (DFCNet), a framework that balances accuracy and speed by performing full feature extraction only on sparse keyframes. For intermediate frames, DFCNet utilizes optical flow to propagate the target state, significantly reducing redundant computations. An adaptive keyframe scheduling mechanism, based on temporal intervals and tracking confidence, ensures robustness. Evaluated on OTB2013 and OTB2015 benchmarks, DFCNet achieves a 60% speedup over the MDNet baseline while maintaining high tracking precision.

Introduction

Visual tracking—following an object through a video stream—is a foundational task for robotics, surveillance, and AR/VR. Modern trackers based on CNNs, such as MDNet, are incredibly robust because they learn rich feature representations. However, these models are computationally intensive. Running a deep CNN on every single frame is often redundant because consecutive frames in a video are usually very similar. The authors of DFCNet sought a way to retain the discriminative power of deep trackers while eliminating unnecessary processing on "easy" frames.

Methods: The Collaborative Architecture

The authors propose splitting the tracking task into two distinct paths within the Deep Flow Collaborative Network (DFCNet):

  1. The Keyframe Branch: When a frame is designated as a "keyframe," the system runs a full feature extraction and an observation model to precisely locate the target. This ensures the tracker stays anchored to the target's current appearance.
  2. The Non-Keyframe Branch: For intermediate frames, the system skips the heavy CNN layers. Instead, it uses a Flow Sub-network (based on FlowNet2) to calculate the optical flow between the current and previous frames. It then "warps" or propagates the previous target state to the current frame using this flow map.
The DFCNet architecture showing the decision logic between keyframe and non-keyframe processing paths.

Adaptive Keyframe Scheduling

The core innovation is the adaptive trigger mechanism. DFCNet decides to run the heavy CNN based on two criteria:

  • Temporal Interval: A fixed interval (e.g., every 3rd frame) ensures periodic re-alignment.
  • Confidence Threshold: If the tracking confidence score drops below a certain level, the system realizes it might be losing the target and immediately triggers a keyframe update to "re-center" itself.

Key Results and Evaluation

The model was evaluated on the OTB2013 and OTB2015 benchmarks, which feature challenges like illumination changes, occlusions, and background clutter.

  • Efficiency: DFCNet-10 achieved a 60% speedup over the baseline MDNet, increasing throughput from ~1.84 frames per second (fps) to 2.95 fps.
  • Accuracy: Despite the reduced computation, DFCNet slightly outperformed MDNet on the OTB2015 benchmark, improving the overlap success rate from 65.0% to 65.7%.
Success plots of OPE on the OTB benchmark, demonstrating DFCNet's competitive performance against other state-of-the-art trackers.

Discussion and Conclusion

DFCNet proves that re-extracting deep features for every frame is often unnecessary. By using optical flow as a "cheap" bridge between expensive computations, the tracker acts intelligently—cruising on "autopilot" when the scene is stable and "waking up" for full processing when the target appearance changes or confidence drops.

While 3 fps is a significant improvement, it remains below the 30+ fps required for real-time fluid applications. Future work may involve integrating more lightweight feature extractors or more efficient flow architectures to push this collaborative approach toward true real-time performance.

Summary updated

BibTeX
@inproceedings{liu2020deep,
  author = {Peidong Liu and Xiyu Yan and Yong Jiang and Shu-Tao Xia},
  title = {{Deep Flow Collaborative Network for Online Visual Tracking}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9054590},
}
Figure