Common Corruptions for Enhancing and Evaluating Robustness in Air-to-Air Visual Object Detection

Anastasios Arsenos, Vasileios Karampinis, Evangelos Petrongonas, Christos Skliros, Dimitrios Kollias, Stefanos D. Kollias, Athanasios Voulodimos

TL;DR

This paper introduces AOT-C, the first benchmark dataset for evaluating the robustness of air-to-air visual object detection models against common corruptions like adverse weather and sensor noise. They found that YOLO-family detectors are more robust than transformer-based models, and finetuning with synthetic corruptions improves real-world performance.

What Problem Does This Paper Address?

Autonomous flight, especially for tasks like navigation and collision avoidance in busy airspace, hinges on reliable visual object detection. A critical challenge is detecting other aircraft (non-cooperative traffic) in real-world conditions, which are far from ideal. Environmental factors like fog, rain, and low light, along with sensor imperfections, can significantly degrade the performance of deep learning models. Existing datasets often lack these realistic corruptions, making it hard to assess how well current detectors will perform in actual flight scenarios. This gap is particularly concerning because failures in aerial object detection can have severe consequences, including mid-air collisions.

Methods

The core of this work involves creating a new dataset and conducting extensive evaluations.

  • Dataset Creation (AOT-C): The authors took the existing Airborne Object Tracking (AOT) dataset and introduced seven types of "common corruptions" to its test set. These corruptions are categorized into weather (fog, rain, low light), sensor noise (color quantization, ISO noise), and defocus. Each corruption is applied at four different severity levels, creating a comprehensive testbed for robustness evaluation.
  • Experimental Evaluation: They evaluated eight diverse object detection models, including popular one-stage detectors (YOLO family, RetinaNet) and multi-stage/transformer-based detectors (Faster R-CNN, DiffusionDet, DETR, CenterNet2).
  • Finetuning Strategy: To demonstrate a practical solution, they explored finetuning models on their synthetically corrupted data and then evaluated their performance on both the original AOT dataset and real-world flight data.

Datasets

  • Airborne Object Tracking (AOT) Dataset: Used as the base for creating the AOT-C benchmark. It's described as the largest and most comprehensive dataset for air-to-air object detection and tracking available at the time of publication.
  • AOT-C (Airborne Object Tracking - Corrupted): The newly introduced benchmark dataset. It's derived from the AOT dataset by applying seven types of synthetic corruptions at four severity levels. This is presented as the first robustness benchmark specifically for air-to-air aerial object detection.
  • Real-world Flight Tests: The authors also conducted evaluations on a custom dataset of 10,641 images captured from two different drones in real-world flight conditions, including adverse weather and varying lighting.

Key Insights and Ideas

  • YOLO Family Dominance: One-stage detectors, particularly those from the YOLO family (YOLOv5, YOLOX), consistently show better robustness against common corruptions compared to more complex transformer-based or multi-stage detectors.
  • Transformer Vulnerability: Models like Faster R-CNN and transformer-based detectors (e.g., DETR) are highly susceptible to corruptions, experiencing significant performance drops.
  • Robustness = Generalization: The study strongly suggests that a model's robustness to corruptions is directly linked to its generalization ability. Models that perform well on corrupted data also tend to generalize better to unseen, real-world scenarios.
  • Synthetic Data for Real-World Gains: Finetuning models on synthetically generated corruptions significantly improves their performance on real-world flight data, especially in challenging conditions. This highlights the practical utility of synthetic data for enhancing robustness.
  • Sensor Noise is a Major Hurdle: While weather conditions have a noticeable impact, sensor noise (especially ISO noise) and defocus corruptions cause the most severe degradation in detection accuracy across all evaluated models.

Key Results

  • Overall Robustness: The average robustness performance (APcor) across all detectors was 22.7% lower than their clean accuracy (APclean), underscoring the widespread issue of performance degradation.
  • Top Performers: YOLOv5 achieved the highest robustness (APcor = 53.5%), while RetinaNet performed the worst (APcor = 20.0%). Among multi-stage networks, DiffusionDet was the most robust (APcor = 35.7%).
  • Finetuning Impact: Finetuning YOLOv5 on synthetic corruptions led to an 11.3% increase in AP on real-world flight tests compared to the base model.
  • Qualitative Improvements: Visual examples showed that finetuned models reduced false positives and missed detections in challenging real-world scenarios (e.g., cloudy-mild rain weather) compared to base models.

Limitations and Future Work

  • Synthetic vs. Real Corruptions: While the authors validate that their synthetic weather corruptions align with real-world performance, they acknowledge that real-world corruptions can be more complex, arising from multiple simultaneous factors.
  • Exhaustiveness of Corruptions: The AOT-C dataset includes seven representative corruptions, but the authors encourage future work to incorporate a more diverse range of challenging conditions.
  • Model Architectures: The study evaluated eight detectors; future work could explore a wider array of architectures and their specific vulnerabilities.
  • Future Directions: The authors suggest that their work lays the groundwork for advancing robust air-to-air aerial object detection and that their findings can guide practitioners in selecting appropriate models. They also emphasize the need for continued efforts to test model vulnerabilities before deployment.

Summary updated

Abstract

The main barrier to achieving fully autonomous flights lies in autonomous aircraft navigation. Managing non-cooperative traffic presents the most important challenge in this problem. The most efficient strategy for handling non-cooperative traffic is based on monocular video processing through deep learning models. This study contributes to the vision-based deep learning aircraft detection and tracking literature by investigating the impact of data corruption arising from environmental and hardware conditions on the effectiveness of these methods. More specifically, we designed $7$ types of common corruptions for camera inputs taking into account real-world flight conditions. By applying these corruptions to the Airborne Object Tracking (AOT) dataset we constructed the first robustness benchmark dataset named AOT-C for air-to-air aerial object detection. The corruptions included in this dataset cover a wide range of challenging conditions such as adverse weather and sensor noise. The second main contribution of this letter is to present an extensive experimental evaluation involving $8$ diverse object detectors to explore the degradation in the performance under escalating levels of corruptions (domain shifts). Based on the evaluation results, the key observations that emerge are the following: 1) One-stage detectors of the YOLO family demonstrate better robustness, 2) Transformer-based and multi-stage detectors like Faster R-CNN are extremely vulnerable to corruptions, 3) Robustness against corruptions is related to the generalization ability of models. The third main contribution is to present that finetuning on our augmented synthetic data results in improvements in the generalisation ability of the object detector in real-world flight experiments.
BibTeX
@article{arsenos2024common,
  author = {Anastasios Arsenos and Vasileios Karampinis and Evangelos Petrongonas and Christos Skliros and Dimitrios Kollias and Stefanos D. Kollias and Athanasios Voulodimos},
  title = {{Common Corruptions for Enhancing and Evaluating Robustness in Air-to-Air Visual Object Detection}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  url = {https://openreview.net/forum?id=lJvftuCEkW},
}
Figure