Rotation Adaptive Visual Object Tracking with Motion Consistency

Litu Rout, Sidhartha, Gorthi R. K. S. S. Manyam, Deepak Mishra

Abstract: Visual object tracking has seen significant advances through the tracking-by-detection paradigm, particularly with the integration of deep convolutional neural networks. However, standard trackers often rely solely on correlation or feature selection, frequently neglecting physical motion constraints. This paper investigates the integration of rotation adaptiveness and motion consistency into high-speed tracking frameworks. By capturing physical constraints, the proposed approach improves accuracy, robustness, and rotation adaptiveness across challenging visual sequences.

1. Introduction

In modern computer vision, "tracking by detection" using architectures like Siamese Fully Convolutional Networks (SiameseFC) and Correlation Filter Networks (CFnet) has become the standard due to their high processing speeds. Despite their efficiency, these trackers possess a significant "blind spot": they typically assume targets remain axis-aligned. When an object undergoes rotation, severe deformation, or erratic movement, these trackers often drift, failing to maintain an accurate bounding box.

Comparison of standard tracking (top) vs. the proposed rotation-adaptive tracking (bottom) on the 'glove' sequence.

Furthermore, most current methods ignore the physical constraints of motion—the principle that objects generally follow smooth, predictable trajectories. This paper addresses these gaps by introducing a framework for Rotation Adaptive Tracking and Motion Consistency, enforcing smoothness in both direction and distance to prevent the tracker from making physically impossible "jumps" caused by background noise.

2. Proposed Methodology

The authors propose a generic approach to enhance existing state-of-the-art trackers, specifically creating "DSR" (Displacement, Scale, and Rotation) versions of SiameseFC and CFnet.

2.1 Base Architecture

The foundation of the approach relies on the Siamese network paradigm, which compares a target exemplar ($z$) against a search image ($x$) to generate a response map. The proposed DSR framework modifies this pipeline to handle geometric deformations.

The base SiameseFC architecture used as a foundation for the DSR modifications.

2.2 Rotation Invariance (RI)

To handle spinning objects, the Siamese DSR tracker extracts features from multiple rotated versions of the target template. It utilizes a Gaussian-weighted average of the nearest neighbor response maps to identify the optimal orientation. For CFnet DSR, which updates every frame, the model applies forward and backward rotations after each update to account for gradual orientation shifts without requiring a massive bank of pre-rotated templates.

2.3 Motion and Scale Consistency

Instead of simply selecting the highest score in a response map—a method prone to noise—the algorithm enforces consistency across frames:

  • Displacement Consistency: The tracker maintains a rolling average of the motion vector's angle and distance. By assigning a small weight (1%) to the previous motion vector, the system acts as a regularizer, ensuring the trajectory remains physically plausible.
  • Scale Consistency: Rather than picking a single winning scale from a pyramid, the system computes a Gaussian weighted average of response maps across different scales, stabilizing the bounding box and reducing jitter.

3. Experimental Results

The DSR-enhanced trackers were evaluated on major industry benchmarks, including OTB (Object Tracking Benchmark) and VOT (Visual Object Tracking) 2016, focusing on sequences with extreme rotations like glove and fish1.

  • VOT 2016 Performance: The Siamese DSR achieved a 15.7% improvement in robustness rank and a 14.3% improvement in accuracy rank over the baseline SiameseFC.
  • OTB Performance: On OTB100, the success rate (AUC) improved by 4.6%, while precision increased by 6.75%.
  • Optimization: Empirical testing revealed that checking for rotations at 8-degree intervals provided the optimal balance between tracking accuracy and computational overhead.

4. Conclusion

By treating physics as a regularizer, the proposed framework significantly reduces tracker drift in the presence of background clutter and geometric changes. While current evaluation toolkits often penalize rotation estimation due to their reliance on axis-aligned bounding boxes, the performance gains observed here demonstrate the clear value of rotation adaptiveness. Future work will explore integrating these motion consistency techniques into deeper architectures like MDNet and TCNN to further refine feature extraction in rotation-invariant branches.

Summary updated

Abstract

Visual Object tracking research has undergone significant improvement in the past few years. The emergence of tracking by detection approach in tracking paradigm has been quite successful in many ways. Recently, deep convolutional neural networks have been extensively used in most successful trackers. Yet, the standard approach has been based on correlation or feature selection with minimal consideration given to motion consistency. Thus, there is still a need to capture various physical constraints through motion consistency which will improve accuracy, robustness and more importantly rotation adaptiveness. Therefore, one of the major aspects of this paper is to investigate the outcome of rotation adaptiveness in visual object tracking. Among other key contributions, the paper also includes various consistencies that turn out to be extremely effective in numerous challenging sequences than the current state-of-the-art.
BibTeX
@inproceedings{rout2018rotation,
  author = {Litu Rout and Sidhartha and Gorthi R. K. S. S. Manyam and Deepak Mishra},
  title = {{Rotation Adaptive Visual Object Tracking with Motion Consistency}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  url = {https://openreview.net/forum?id=h7B8-bijWvrS},
}
Figure