Fine-Grained Motion Representation For Template-Free Visual Tracking

Kai Shuang, Yuheng Huang, Yue Sun, Zhun Cai, Hao GuoWACV 2020

TL;DR

This paper introduces FGTrack, a novel template-free visual tracking framework that moves beyond simple displacement estimation. By leveraging dense optical flow, FGTrack estimates pixel-level transformations including displacement, scaling, and rotation, leading to more accurate and robust object tracking. This fine-grained motion representation, combined with an appearance feature branch and a novel loss function, achieves state-of-the-art performance on challenging benchmarks.

What Problem Does This Paper Address?

Generic visual object tracking aims to follow an object across video frames, which is notoriously difficult due to variations in scale, deformation, and illumination. Traditional template-based trackers often struggle with generalization and require online model updates. Template-free approaches, while aiming for better generalization by avoiding explicit template learning, have historically been limited by their coarse-grained motion representation. They typically only estimate the object's displacement, which is insufficient to capture complex transformations like scaling and rotation. This limitation hinders their accuracy, forcing them to rely on template-based discriminative models as a workaround, thus undermining the template-free paradigm's core principle. This paper addresses this gap by proposing a method to extract richer, pixel-level motion information from optical flow to represent these complex transformations.

Methods

FGTrack is built upon a template-free paradigm that leverages dense optical flow. The core innovation lies in its fine-grained motion estimation. Instead of just estimating the bounding box displacement, FGTrack's movement estimation branch predicts three parameters for each pixel:

  • Displacement: Directly inferred from the optical flow vector field.
  • Scaling: Estimated using the divergence of the optical flow field, which quantifies the "flux" of pixels passing through a bounding box boundary.
  • Rotation: Estimated using the curl of the optical flow field, representing the rotational movement of pixels around a path.

These pixel-level transformations are then aggregated by an object-level reducer to estimate the overall bounding box transformation.

To further enhance accuracy and robustness, FGTrack incorporates an auxiliary appearance feature branch. This branch uses a pre-trained feature extractor (ResNet) and a spatial attention mechanism to guide the object-level reducer to focus on more reliable, feature-rich regions, thereby improving localization without sacrificing generalization.

A novel Limited L1 Loss function is also introduced to mitigate the impact of optical flow estimation failures during training. This loss function suppresses gradients from outliers, allowing the model to focus on learning from crucial, accurate motion information.

Datasets

The paper uses the following datasets:

  • Training: TrackingNet, ImageNet-VID, and YouTube-VOS (for training the rotation head, using minimum bounding rectangles of binary masks).
  • Evaluation: NFS (Need For Speed) and VOT2017.

Key Insights and Ideas

  • Optical Flow as a Richer Motion Cue: The paper convincingly argues that dense optical flow contains more information than just displacement. By applying differential operators (divergence and curl), it's possible to extract scaling and rotation information at a pixel level.
  • Pixel-Level Transformations for Robustness: Estimating motion at the pixel level, rather than just the object level, provides a more detailed and potentially more robust representation of how an object deforms and moves.
  • Appearance as an Auxiliary for Generalization: Integrating an appearance branch as an auxiliary component, rather than the primary feature, is a smart way to leverage appearance information for accuracy without compromising the template-free model's generalization capabilities.
  • Limited L1 Loss for Training Stability: Optical flow estimators are known to fail in challenging scenarios. The Limited L1 Loss is a practical solution to prevent these failures from derailing the training process.

Key Results

FGTrack achieves state-of-the-art performance on several benchmarks:

  • NFS: Achieves an AUC score of 50.6, outperforming BACF by 4% and other state-of-the-art trackers like DaSiamRPN, C-COT, and ECO.
  • VOT2017: Shows competitive performance, slightly outperforming AMNet on the EAO metric.
  • Speed: The tracker runs at an average speed of 19.6 FPS on a NVIDIA GeForce 1080Ti GPU.
  • Ablation Studies: Demonstrate the significant impact of each component: the Limited L1 Loss (+Loss), rotated bounding box estimation (+R), bounding box regression (+BBR), and spatial attention (+SA) all contribute to improved EAO scores. The full FGTrack model achieves an EAO of 0.287.

Limitations and Future Work

  • Dependence on Optical Flow Quality: The paper acknowledges that the performance of FGTrack is inherently limited by the quality of the optical flow estimator. While they use FlowNet2 and PWC-Net, they suggest that a more powerful optical flow estimator could further improve results.
  • Generalization to Extreme Cases: While the Limited L1 Loss helps with outliers, extremely challenging scenarios like rapid scene changes or objects with very few features might still pose difficulties.
  • Future Work: The authors suggest exploring more powerful optical flow estimators and integrating FGTrack with other tracking paradigms, such as Siamese trackers, to further enhance performance. They also highlight the potential applicability of their fine-grained motion representation in other computer vision tasks that utilize optical flow.

Summary updated

BibTeX
@inproceedings{shuang2020fine,
  author = {Kai Shuang and Yuheng Huang and Yue Sun and Zhun Cai and Hao Guo},
  title = {{Fine-Grained Motion Representation For Template-Free Visual Tracking}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure