Abstract
We address the problem of human action classification in drone videos. Due to the high cost of capturing and labeling large-scale drone videos with diverse actions, we present unsupervised and semi-supervised domain adaptation approaches that leverage both the existing fully annotated action recognition datasets and unannotated (or only a few annotated) videos from drones. To study the emerging problem of drone-based action recognition, we create a new dataset, NEC-DRONE, containing 5,250 videos to evaluate the task. We tackle both problem settings with 1) same and 2) different action label sets for the source (e.g., Kinetics dataset) and target domains (drone videos). We present a combination of video and instance-based adaptation methods, paired with either a classifier or an embedding-based framework to transfer the knowledge from source to target. Our results show that the proposed adaptation approach substantially improves the performance on these challenging and practical tasks.
The Challenge of Drone Action Recognition
Human action recognition has achieved significant success using large-scale datasets like Kinetics or UCF-101. However, these datasets mostly consist of "third-person" videos—footage taken by humans standing on the ground. Drones present a massive domain gap: they fly at varying heights and angles, move continuously (causing motion blur and jitters), and the subjects (humans) often appear much smaller and off-center compared to standard datasets.
Manually labeling thousands of drone videos is expensive and time-consuming. This research asks: Can we train a model on existing ground-level datasets and make it work on drone footage without needing a massive new labeled dataset?
The NEC-DRONE Dataset
To address this, the authors introduced NEC-DRONE, a large-scale dataset containing 5,250 videos of 19 actors performing 16 different actions, such as "shaking hands" and "leaving a backpack." This dataset highlights the unique perspectives and challenges inherent in aerial surveillance and drone-based monitoring.
Methodology
The proposed framework utilizes the I3D (Inflated 3D ConvNet) architecture as a backbone and employs two primary strategies for domain alignment:
- Adversarial Domain Adaptation: A domain classifier with a Gradient Reversal Layer (GRL) forces the network to learn features that are useful for recognizing actions but invariant to the camera viewpoint (ground vs. drone).
- Metric Learning (Triplet Loss): For scenarios where the source and target datasets have different label sets, the authors use triplet loss to create an embedding space where similar actions cluster together regardless of the perspective.
- Video vs. Instance-based Adaptation: While video-based adaptation processes the whole frame, instance-based adaptation uses a person detector to crop "human tubes." This focuses the model on the actor's motion, ignoring background noise and drone ego-motion. A "late-fusion" combination of both approaches yielded the best results.
Key Results and Insights
The proposed adaptation approach demonstrated substantial gains over baseline models:
- Unsupervised Domain Adaptation (UDA): A model trained only on Kinetics (Source only) achieved just 13.6% accuracy on NEC-DRONE. Applying UDA pushed this to 32.0%.
- Semi-Supervised Learning: By adding just 20 labeled drone examples per class, accuracy jumped to 58.3%, significantly closing the gap toward the fully supervised upper bound of 76.7%.
- Cross-View Success: The method also proved effective on the Charades-Ego dataset, outperforming existing "Actor and Observer" models without requiring paired video data.
Conclusion
While a performance gap remains between domain-adapted and fully supervised models, this research provides a robust framework for leveraging existing ground-level data for drone applications. The combination of instance-based "human tubes" and adversarial alignment offers a practical path forward for action recognition in the increasingly important field of drone-based computer vision.
Summary updated