Drone Shadow Tracking

Xiaoyan Zou, Ruofan Zhou, Majed El Helou, Sabine SüsstrunkarXiv 2019 · arXiv

Abstract

Tracking a drone's own shadow in aerial footage is a critical prerequisite for high-quality shadow removal and inpainting. However, shadows are challenging targets due to constant shape deformation and loss of contrast when passing over dark objects. This paper proposes a robust tracking framework that integrates traditional correlation filters with physical shadow detection masks. By implementing a failure prediction mechanism and a "freeze" strategy during high-uncertainty periods, the system prevents tracker drift and enables recovery in complex environments. Experimental results on a custom dataset demonstrate that our method achieves superior robustness and real-time performance compared to both traditional and deep learning-based trackers.

Introduction

When a drone flies at low to medium altitudes, its shadow often appears in the frame, creating a distracting visual artifact for professional cinematography. To remove this shadow via inpainting, it must first be tracked accurately across every frame.

However, drone shadows represent a unique challenge for computer vision. They undergo constant deformation as the drone rotates or changes altitude, and their geometry morphs based on the 3D structure of the ground. Most critically, shadows often perform "vanishing acts" when passing over dark surfaces, such as asphalt or black backpacks. In these high-contrast environments, standard trackers often lose the target and "drift" toward the dark object itself.

Comparison of tracking performance. (b) Standard MOSSE tracking drifts to a dark backpack, whereas (c) the proposed method maintains focus on the shadow.

Methodology

The authors propose a hybrid approach that fuses high-speed Kernel-based tracking with Physical shadow properties to mitigate drift.

Core Components

  1. Base Tracker (MOSSE): The system utilizes the Minimum Output Sum of Squared Error (MOSSE) filter, a high-speed algorithm operating in the Fourier domain to find target matches.
  2. Shadow Detection Mask: Instead of relying solely on geometric features, the system uses adaptive thresholding to create a binary mask of potential shadow pixels based on local intensity.
  3. Failure Prediction: The algorithm monitors the detected shadow area. If the area suddenly increases (e.g., $A_{i+1} \ge 2.5 \times A_i$), the system predicts a tracking failure, assuming the tracker has "glued" itself to a different dark object.
  4. Shadow Fusion Tracking: Upon failure prediction, the system expands the search area by 33% and freezes the kernel update to prevent the tracker from learning the wrong object. It then performs pixel-wise multiplication between the correlation map and the physical shadow mask to filter out non-shadow candidates.

Experimental Evaluation

The researchers evaluated the method using a custom dataset of 8 drone videos ($1280 \times 720$ resolution) captured with a DJI Mavic Air. The dataset includes:

  • Simple Scenarios: Shadows on flat ground, paper bags, or newspapers.
  • Difficult Scenarios: Shadows moving over black laptops, backpacks, textured grass, and moving people.

Performance was measured using the Intersection over Union (IoU) metric against manually annotated ground truth for 25 frames around each failure case.

Results and Discussion

The proposed method demonstrates significant improvements in robustness and recovery. In difficult scenes where the baseline MOSSE tracker failed completely and the state-of-the-art deep learning tracker (DAT) failed 50% of the time, the proposed method maintained a consistent lock.

Success rate vs. overlap threshold. The proposed method (yellow) significantly outperforms MOSSE (blue) and the deep-learning-based DAT (red) tracker.

Key Insights

  • Physics over Geometry: For shadows, physical properties (how they affect light) are more reliable than their ever-changing shapes.
  • The "Freeze" Strategy: By freezing the model during high-uncertainty moments, the tracker can effectively "wait" for the shadow to reappear on a lighter surface rather than adapting to the background.
  • Efficiency: While deep learning trackers like DAT required ~4.8 seconds per frame on a Titan X GPU, this correlation-filter approach is designed for real-time applications on standard drone hardware.

Conclusion

By combining traditional correlation filters with physical shadow detection, this method provides a robust solution for drone shadow tracking in complex environments. Future work may involve integrating Near-Infrared (NIR) sensors to improve initial shadow detection and addressing complex 3D geometries that cause shadows to break into disconnected pieces.

Summary updated

Abstract

Aerial videos taken by a drone not too far above the surface may contain the drone's shadow projected on the scene. This deteriorates the aesthetic quality of videos. With the presence of other shadows, shadow removal cannot be directly applied, and the shadow of the drone must be tracked. Tracking a drone's shadow in a video is, however, challenging. The varying size, shape, change of orientation and drone altitude pose difficulties. The shadow can also easily disappear over dark areas. However, a shadow has specific properties that can be leveraged, besides its geometric shape. In this paper, we incorporate knowledge of the shadow's physical properties, in the form of shadow detection masks, into a correlation-based tracking algorithm. We capture a test set of aerial videos taken with different settings and compare our results to those of a state-of-the-art tracking algorithm.
BibTeX
@misc{zou2019drone,
  author = {Xiaoyan Zou and Ruofan Zhou and Majed El Helou and Sabine Süsstrunk},
  title = {{Drone Shadow Tracking}},
  year = {2019},
  eprint = {1905.08214},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.08214},
}
Figure