TL;DR
This paper introduces AOT-C, the first benchmark dataset for evaluating the robustness of air-to-air visual object detection models against common corruptions like adverse weather and sensor noise. They found that YOLO-family detectors are more robust than transformer-based models, and finetuning with synthetic corruptions improves real-world performance.
What Problem Does This Paper Address?
Autonomous flight, especially for tasks like navigation and collision avoidance in busy airspace, hinges on reliable visual object detection. A critical challenge is detecting other aircraft (non-cooperative traffic) in real-world conditions, which are far from ideal. Environmental factors like fog, rain, and low light, along with sensor imperfections, can significantly degrade the performance of deep learning models. Existing datasets often lack these realistic corruptions, making it hard to assess how well current detectors will perform in actual flight scenarios. This gap is particularly concerning because failures in aerial object detection can have severe consequences, including mid-air collisions.
Methods
The core of this work involves creating a new dataset and conducting extensive evaluations.
- Dataset Creation (AOT-C): The authors took the existing Airborne Object Tracking (AOT) dataset and introduced seven types of "common corruptions" to its test set. These corruptions are categorized into weather (fog, rain, low light), sensor noise (color quantization, ISO noise), and defocus. Each corruption is applied at four different severity levels, creating a comprehensive testbed for robustness evaluation.
- Experimental Evaluation: They evaluated eight diverse object detection models, including popular one-stage detectors (YOLO family, RetinaNet) and multi-stage/transformer-based detectors (Faster R-CNN, DiffusionDet, DETR, CenterNet2).
- Finetuning Strategy: To demonstrate a practical solution, they explored finetuning models on their synthetically corrupted data and then evaluated their performance on both the original AOT dataset and real-world flight data.
Datasets
- Airborne Object Tracking (AOT) Dataset: Used as the base for creating the AOT-C benchmark. It's described as the largest and most comprehensive dataset for air-to-air object detection and tracking available at the time of publication.
- AOT-C (Airborne Object Tracking - Corrupted): The newly introduced benchmark dataset. It's derived from the AOT dataset by applying seven types of synthetic corruptions at four severity levels. This is presented as the first robustness benchmark specifically for air-to-air aerial object detection.
- Real-world Flight Tests: The authors also conducted evaluations on a custom dataset of 10,641 images captured from two different drones in real-world flight conditions, including adverse weather and varying lighting.
Key Insights and Ideas
- YOLO Family Dominance: One-stage detectors, particularly those from the YOLO family (YOLOv5, YOLOX), consistently show better robustness against common corruptions compared to more complex transformer-based or multi-stage detectors.
- Transformer Vulnerability: Models like Faster R-CNN and transformer-based detectors (e.g., DETR) are highly susceptible to corruptions, experiencing significant performance drops.
- Robustness = Generalization: The study strongly suggests that a model's robustness to corruptions is directly linked to its generalization ability. Models that perform well on corrupted data also tend to generalize better to unseen, real-world scenarios.
- Synthetic Data for Real-World Gains: Finetuning models on synthetically generated corruptions significantly improves their performance on real-world flight data, especially in challenging conditions. This highlights the practical utility of synthetic data for enhancing robustness.
- Sensor Noise is a Major Hurdle: While weather conditions have a noticeable impact, sensor noise (especially ISO noise) and defocus corruptions cause the most severe degradation in detection accuracy across all evaluated models.
Key Results
- Overall Robustness: The average robustness performance (APcor) across all detectors was 22.7% lower than their clean accuracy (APclean), underscoring the widespread issue of performance degradation.
- Top Performers: YOLOv5 achieved the highest robustness (APcor = 53.5%), while RetinaNet performed the worst (APcor = 20.0%). Among multi-stage networks, DiffusionDet was the most robust (APcor = 35.7%).
- Finetuning Impact: Finetuning YOLOv5 on synthetic corruptions led to an 11.3% increase in AP on real-world flight tests compared to the base model.
- Qualitative Improvements: Visual examples showed that finetuned models reduced false positives and missed detections in challenging real-world scenarios (e.g., cloudy-mild rain weather) compared to base models.
Limitations and Future Work
- Synthetic vs. Real Corruptions: While the authors validate that their synthetic weather corruptions align with real-world performance, they acknowledge that real-world corruptions can be more complex, arising from multiple simultaneous factors.
- Exhaustiveness of Corruptions: The AOT-C dataset includes seven representative corruptions, but the authors encourage future work to incorporate a more diverse range of challenging conditions.
- Model Architectures: The study evaluated eight detectors; future work could explore a wider array of architectures and their specific vulnerabilities.
- Future Directions: The authors suggest that their work lays the groundwork for advancing robust air-to-air aerial object detection and that their findings can guide practitioners in selecting appropriate models. They also emphasize the need for continued efforts to test model vulnerabilities before deployment.
Summary updated