TL;DR
Thermal infrared (IR) drone detection is currently bottlenecked more by inconsistent dataset quality and undocumented sensor artifacts than by model architecture. This benchmark reveals that even state-of-the-art YOLO models fail to detect over 40% of targets, with performance plummeting during "thermal crossover" events and in cluttered urban environments.
What Problem Does This Paper Address?
While visible-spectrum (RGB) cameras are the standard for drone surveillance, they are rendered useless by darkness or fog. Thermal infrared (IR) imaging offers a robust alternative for 24/7 monitoring. However, the field of IR drone detection is currently a "wild west" of inconsistent data.
Public datasets vary wildly in terms of camera sensors (cooled vs. uncooled), image processing (tone mapping, sharpening), and annotation practices. This makes it nearly impossible to tell if a new model is actually better or if it has simply learned to exploit specific artifacts—like on-screen display (OSD) text or sensor-specific noise—unique to a single dataset. The authors aim to systematically characterize these variabilities and establish a rigorous benchmark for cross-dataset generalization.
Methods
The researchers approached the problem through three main pillars:
- Artifact Taxonomy: They established a taxonomy of visual artifacts based on their origin in the imaging pipeline: Optics & Sensor (e.g., motion blur, vignetting), Tone Mapping (e.g., histogram equalization), and Recording (e.g., OSD overlays, compression).
- Model Benchmarking: They evaluated two generations of the popular "You Only Look Once" architecture—YOLOv8 and YOLO11—across various scales (nano to large) to see if architectural improvements could overcome data limitations.
- Cross-Dataset Evaluation: Instead of just testing on a "held-out" portion of the same dataset, they performed cross-domain testing: training on one dataset (e.g., CST Anti-UAV) and testing on entirely different ones (e.g., LRDDv3) to measure real-world robustness.
Datasets
The authors reviewed nine publicly available datasets (summarized in Figure 4/Table 1) and focused their deep-dive analysis on three major Long-Wave Infrared (LWIR) benchmarks:
- Hamburg2 (H2): Features scene-matched subsets from two different cameras (FLIR and InfraTec).
- LRDDv3: An air-to-air dataset with a high proportion of very small targets.
- CST Anti-UAV: A large-scale dataset (238k frames) with diverse backgrounds and detailed attribute labels like thermal crossover.
The datasets show significant differences in target size distributions, with LRDDv3 and CST being dominated by "extra-extra-small" (XXS) targets.
Key Insights and Ideas
- Data Composition Trumps Model Capacity: One of the most striking findings is that detection performance is governed much more strongly by the training data than by the specific YOLO generation or model size. Moving from a "small" to a "large" model yields marginal gains compared to the massive performance swings caused by switching training sets.
- Asymmetric Generalization: Models do not transfer well between datasets. For example, a model trained on the grayscale CST dataset fails almost completely on the pseudocolor-mapped LRDDv3 dataset. This highlights how "baked-in" image processing choices can break a detector in the field.
- The "Thermal Crossover" Trap: Performance drops significantly during thermal crossover (TC)—periods where the drone's temperature matches the background, leading to near-zero contrast. This is a fundamental physical limit that current deep learning models struggle to navigate.
- Shortcut Learning: The authors found evidence that models might be learning "shortcuts." For instance, OSD overlays (text on screen) can act as spurious cues that confuse the model or lead to false positives if the model associates text-heavy regions with targets.
Key Results
- High Miss Rates: Even the best-performing configurations had a False Negative Rate (FNR) exceeding 40%. In critical infrastructure protection, missing 4 out of 10 drones is often unacceptable.
- Small Target Failure: For XXS targets (area < 8² pixels), the miss rate is particularly high, as these targets often fall below the stride of the model's finest feature maps.
- mAP Scores: In-domain [email protected] scores reached up to 0.747 (on CST), but cross-domain scores frequently plummeted below 0.10, illustrating a severe lack of robustness.
- YOLO11 vs. YOLOv8: YOLO11 showed slight improvements in efficiency, achieving comparable mAP to YOLOv8l with significantly fewer parameters and FLOPs, but it did not solve the underlying generalization issues.
Limitations and Future Work
The authors acknowledge that existing datasets are often poorly documented regarding their imaging pipelines (e.g., whether Automatic Gain Control was active). This lack of metadata prevents researchers from properly accounting for domain shifts.
Future Directions: * Synthetic Data: The authors suggest using synthetic data generation to systematically fill gaps in existing datasets, such as specific weather conditions or rare drone models. * Radiometric Data: Moving toward raw radiometric data (temperature values) rather than 8-bit "visualized" IR images could eliminate the biases introduced by tone mapping and pseudocolor schemes. * Standardized Annotations: There is a dire need for consistent bounding-box conventions (e.g., whether to include motion-blurred propellers) to ensure fair evaluation across the community.
Summary updated