Effect of Annotation Errors on Drone Detection with YOLOv3

Aybora Koksal, Kutalmis Gokalp Ince, A. Aydin AlatanWorkshops @ CVPR 2020 · DOI

Abstract

Deep learning models for object detection rely heavily on high-quality "ground truth" labels. In the context of drone detection, where targets are small and fast-moving, annotation errors are common and potentially devastating. This paper quantifies the impact of three types of labeling noise—missing, shifted, and extra bounding boxes—on the YOLOv3 detector. Furthermore, it introduces a semi-automatic correction method based on template matching to refine noisy datasets. Results indicate that while models are resilient to random extra boxes, systematic shifts and missing annotations significantly degrade performance.

Introduction

Detecting and tracking drones presents unique challenges for computer vision. Drones often occupy only a few pixels and must be distinguished from birds or distant aircraft against complex, dynamic backgrounds. High-quality training data is essential, yet video datasets are frequently labeled using semi-automatic tools that introduce errors like box drift or missed detections during occlusions.

Drones in the Anti-UAV dataset across various backgrounds including clear sky, clouds, and urban environments.

This research focuses on the CVPR-2020 Anti-UAV Challenge dataset, specifically the thermal (IR) imagery, which is vital for 24/7 surveillance. The study aims to determine exactly how much these common annotation mistakes hurt model performance compared to traditional benchmarks like the KITTI dataset.

Methodology and Error Simulation

The researchers evaluated the YOLOv3 architecture by intentionally corrupting training data with three synthetic noise types:

  1. Additional Boxes: Randomly placed boxes containing no target, simulating human error or tracker false positives.
  2. Missing Boxes: Random removal of valid annotations to simulate oversight or occlusion.
  3. Shifted Boxes: Adding Gaussian noise to box coordinates to simulate "drift" often seen in tracking-based annotation tools.
Illustration of annotation errors in thermal drone imagery, showing shifted and additional bounding boxes.

To address these errors, the authors implemented a semi-automatic correction method. Using cross-correlation-based template matching, they estimated the "true" center of an object by comparing consecutive frames (at 30fps). A human-in-the-loop then verified these mathematical corrections to ensure dataset integrity.

Key Insights: The Nature of Label Noise

The study revealed that not all noise impacts the model equally:

  • Resilience to Randomness: YOLOv3 is surprisingly robust against random "additional boxes." The network effectively learns to treat these as background noise.
  • The Severity of Missing Data: Missing boxes (false negatives) are far more damaging. They actively teach the model to ignore valid targets, leading to a significant drop in detection sensitivity.
  • The Localization "Achilles' Heel": Systematic "shifted boxes" are the most detrimental. Even minor pixel-level drift causes Intersection over Union (IoU) scores to plummet, which is catastrophic for tracking applications. Persistent drift across frames is particularly confusing, as the model begins to associate incorrect features with the target class.

Experimental Results

The impact of these errors was quantified through several experiments:

  • Baseline: Using the original, slightly noisy annotations, YOLOv3 achieved a 97.5% Hit Rate (HR).
  • Shifted Box Impact: When boxes were shifted by a standard deviation of 10% of the object size, the Hit Rate crashed from 97.5% to 29.9%. This highlights that precise localization is the "Achilles' heel" of drone detection.
  • Missing Data Impact: Removing 25% of the annotations reduced the Hit Rate to 94.1%.
  • Correction Gains: After applying their semi-automatic correction, the authors saw the Hit Rate improve to 98.8% and the Tracking Accuracy (TA) increase from 73.6% to 76.3%.

Conclusion

This research underscores the critical importance of label precision in specialized detection tasks like drone tracking. While "classical" methods like template matching may seem dated, they remain highly effective for refining high-frame-rate video data. Future work could explore integrating these correction algorithms directly into the training loop, enabling models to self-correct labels as they develop a better understanding of target appearances.

Summary updated

BibTeX
@inproceedings{koksal2020effect,
  author = {Aybora Koksal and Kutalmis Gokalp Ince and A. Aydin Alatan},
  title = {{Effect of Annotation Errors on Drone Detection with YOLOv3}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  doi = {10.1109/cvprw50498.2020.00523},
}
Figure