Abstract
Tracking a drone's own shadow in aerial footage is a critical prerequisite for high-quality shadow removal and inpainting. However, shadows are challenging targets due to constant shape deformation and loss of contrast when passing over dark objects. This paper proposes a robust tracking framework that integrates traditional correlation filters with physical shadow detection masks. By implementing a failure prediction mechanism and a "freeze" strategy during high-uncertainty periods, the system prevents tracker drift and enables recovery in complex environments. Experimental results on a custom dataset demonstrate that our method achieves superior robustness and real-time performance compared to both traditional and deep learning-based trackers.
Introduction
When a drone flies at low to medium altitudes, its shadow often appears in the frame, creating a distracting visual artifact for professional cinematography. To remove this shadow via inpainting, it must first be tracked accurately across every frame.
However, drone shadows represent a unique challenge for computer vision. They undergo constant deformation as the drone rotates or changes altitude, and their geometry morphs based on the 3D structure of the ground. Most critically, shadows often perform "vanishing acts" when passing over dark surfaces, such as asphalt or black backpacks. In these high-contrast environments, standard trackers often lose the target and "drift" toward the dark object itself.
Methodology
The authors propose a hybrid approach that fuses high-speed Kernel-based tracking with Physical shadow properties to mitigate drift.
Core Components
- Base Tracker (MOSSE): The system utilizes the Minimum Output Sum of Squared Error (MOSSE) filter, a high-speed algorithm operating in the Fourier domain to find target matches.
- Shadow Detection Mask: Instead of relying solely on geometric features, the system uses adaptive thresholding to create a binary mask of potential shadow pixels based on local intensity.
- Failure Prediction: The algorithm monitors the detected shadow area. If the area suddenly increases (e.g., $A_{i+1} \ge 2.5 \times A_i$), the system predicts a tracking failure, assuming the tracker has "glued" itself to a different dark object.
- Shadow Fusion Tracking: Upon failure prediction, the system expands the search area by 33% and freezes the kernel update to prevent the tracker from learning the wrong object. It then performs pixel-wise multiplication between the correlation map and the physical shadow mask to filter out non-shadow candidates.
Experimental Evaluation
The researchers evaluated the method using a custom dataset of 8 drone videos ($1280 \times 720$ resolution) captured with a DJI Mavic Air. The dataset includes:
- Simple Scenarios: Shadows on flat ground, paper bags, or newspapers.
- Difficult Scenarios: Shadows moving over black laptops, backpacks, textured grass, and moving people.
Performance was measured using the Intersection over Union (IoU) metric against manually annotated ground truth for 25 frames around each failure case.
Results and Discussion
The proposed method demonstrates significant improvements in robustness and recovery. In difficult scenes where the baseline MOSSE tracker failed completely and the state-of-the-art deep learning tracker (DAT) failed 50% of the time, the proposed method maintained a consistent lock.
Key Insights
- Physics over Geometry: For shadows, physical properties (how they affect light) are more reliable than their ever-changing shapes.
- The "Freeze" Strategy: By freezing the model during high-uncertainty moments, the tracker can effectively "wait" for the shadow to reappear on a lighter surface rather than adapting to the background.
- Efficiency: While deep learning trackers like DAT required ~4.8 seconds per frame on a Titan X GPU, this correlation-filter approach is designed for real-time applications on standard drone hardware.
Conclusion
By combining traditional correlation filters with physical shadow detection, this method provides a robust solution for drone shadow tracking in complex environments. Future work may involve integrating Near-Infrared (NIR) sensors to improve initial shadow detection and addressing complex 3D geometries that cause shadows to break into disconnected pieces.
Summary updated