Cross-View Contextual Relation Transferred Network for Unsupervised Vehicle Tracking in Drone Videos

Wenfeng Song, Shuai Li, Tao Chang, Aimin Hao, Qinping Zhao, Hong QinWACV 2020 · OpenReview

Abstract

Recently CNN-centric object tracking methods have been gaining tremendous success in ground-view videos, however, it remains hard to cope with vehicle tracking in unmanned aerial vehicle (UAV) videos. The key difficulties mainly stem from lacking large-scale well-labeled training datasets and view-invariant appearance model for fast-moving drone-view vehicles. We enhance the vehicle's cross-view feature by exploring relations between the pivotal context and the target to facilitate unsupervised vehicle tracking. The relation is modeled as the relevance of the target and its contextual regions in the tracking task. Specifically, we propose a contextual relation actor-critic (CRAC) framework integrates an actor-critic agent with a dual GAN learning mechanism, which aims to dynamically search the related contextual regions and transfer the relations from ground-view to drone-view videos while retaining the discriminative features. We demonstrate that CRAC could be applied to several state-of-the-art trackers by extensive experiments and ablation studies on four public benchmarks. All the experiments confirm that, our CRAC can improve the performance of state-of-the-art methods in terms of accuracy, robustness, and versatility.

Introduction

Vehicle tracking from Unmanned Aerial Vehicles (UAVs) is a critical component for modern surveillance, traffic management, and search-and-rescue operations. However, standard tracking algorithms trained on ground-level datasets (like dashcam footage) often fail when applied to the top-down perspective of a drone. This performance gap is primarily due to the scarcity of large-scale labeled drone-view datasets and the extreme appearance variations caused by drastic changes in camera angle and altitude.

While a vehicle's visual appearance changes significantly between a street-level view and an aerial view, its contextual relations—such as its position relative to road lanes, intersections, or other vehicles—remain relatively stable. The "Cross-View Contextual Relation Transferred Network" (CRAC) addresses the tracking challenge by transferring these stable relations from well-labeled ground-view datasets to unlabeled drone-view data in an unsupervised manner.

The CRAC Framework

The CRAC framework is designed to bridge the domain gap by focusing on relational features rather than raw pixel appearance. It consists of two primary pillars: Contextual Relation Search and Cross-View Contextual Relation Transfer.

Contextual Relation Search (Actor-Critic)

Rather than relying on a fixed-size bounding box for context, CRAC employs a reinforcement learning agent. The Actor component of the agent dynamically adjusts the context window by choosing to enlarge, shrink, or terminate the search. This allows the system to capture useful environmental clues (like road boundaries) when the target is ambiguous, while excluding noisy distractors (like trees or buildings) that could lead to tracking drift.

Cross-View Contextual Relation Transfer (Dual GAN)

To adapt features across views, the framework utilizes a dual Generative Adversarial Network (GAN) mechanism:

  1. T-GAN (Tracking-guided GAN): This adapts ground-view images to match the visual characteristics of drone-view footage (e.g., lower resolution and different aspect ratios) while ensuring that tracking-relevant features remain discriminative.
  2. A-GAN (Attention GAN): This generates attention maps that highlight critical global relations, further refining the local features provided by the T-GAN.
The CRAC framework transfers contextual relations from ground-view (top) to drone-view (bottom) to improve tracking robustness.

This "plug-and-play" architecture can be wrapped around existing high-performance trackers, such as MDNET or SiamRPN, to enhance their capabilities in aerial scenarios without requiring manual labels for the drone domain.

Experimental Results

The CRAC framework was evaluated using labeled ground-view data from KITTI and VOT13-15 for training, and several drone-view benchmarks (Drone2018, UAV123, DTB, and UAV-Vehicle) for unsupervised adaptation and testing. Integration with baseline trackers showed significant improvements:

  • CRAC-mdnet: Improved the precision of the baseline MDNET on the Drone2018 dataset from 66.7% to 72.8%.
  • CRAC-siam: Maintained a real-time processing speed of 56 FPS while achieving top-tier accuracy among compared methods.

The system demonstrated particular strength in handling challenging scenarios such as fast motion, illumination changes, and partial occlusions. By prioritizing stable contextual relations over volatile appearance features, CRAC provides a robust solution for unsupervised domain adaptation in aerial tracking.

Conclusion and Future Work

The CRAC framework successfully demonstrates that transferring contextual relations is a viable strategy for unsupervised tracking in drone videos. By combining reinforcement learning for dynamic context search with a dual GAN for domain adaptation, the method significantly boosts the performance and versatility of existing trackers.

Future research will focus on addressing full occlusion, where a vehicle is hidden for extended periods under bridges or thick canopies. Incorporating temporal constraints and historical tracking data may provide the necessary continuity to maintain lock-on in these complex environments.

Summary updated

Abstract

Recently CNN-centric object tracking methods have been gaining tremendous success in ground-view videos, however, it remains hard to cope with vehicle tracking in unmanned aerial vehicle (UAV) videos. The key difficulties mainly stem from lacking large-scale well-labeled training datasets and view-invariant appearance model for fast-moving drone-view vehicles. We enhance the vehicle's cross-view feature by exploring relations between the pivotal context and the target to facilitate unsupervised vehicle tracking. The relation is modeled as the relevance of the target and its contextual regions in the tracking task. Specifically, we propose a contextual relation actor-critic (CRAC) framework integrates an actor-critic agent with a dual GAN learning mechanism, which aims to dynamically search the related contextual regions and transfer the relations from ground-view to drone-view videos while retaining the discriminative features. We demonstrate that CRAC could be applied to several state-of-the-art trackers by extensive experiments and ablation studies on four public benchmarks. All the experiments confirm that, our CRAC can improve the performance of state-of-the-art methods in terms of accuracy, robustness, and versatility.
BibTeX
@inproceedings{song2020cross,
  author = {Wenfeng Song and Shuai Li and Tao Chang and Aimin Hao and Qinping Zhao and Hong Qin},
  title = {{Cross-View Contextual Relation Transferred Network for Unsupervised Vehicle Tracking in Drone Videos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=2JWLL-qW4Tw},
}
Figure