Common Corruptions for Evaluating and Enhancing Robustness in Air-to-Air Visual Object Detection

Anastasios Arsenos, Vasileios Karampinis, Evangelos Petrongonas, Christos Skliros, Dimitrios Kollias, Stefanos D. Kollias, Athanasios Voulodimos

TL;DR

This paper introduces AOT-C, the first benchmark dataset for evaluating the robustness of air-to-air visual object detection models against common corruptions like adverse weather and sensor noise. They found that YOLO-family detectors are more robust than transformer-based models, and that fine-tuning with synthetic corruptions improves real-world performance.

What Problem Does This Paper Address?

Autonomous flight, especially in busy low-altitude airspace, relies heavily on aircraft being able to "see and avoid" other objects. This is particularly challenging for air-to-air detection, where one aircraft needs to identify others using its own cameras. Deep learning models have shown great promise in this area, but their performance can degrade significantly when faced with real-world conditions like fog, rain, or sensor noise. These "corruptions" can cause domain shifts, where the data the model encounters in deployment differs from its training data, leading to unpredictable failures. The paper aims to address this by systematically evaluating and improving the robustness of these critical detection systems.

Methods

The core of the paper's methodology involves two main parts:

  1. Creating a Robustness Benchmark (AOT-C): The authors took the existing Airborne Object Tracking (AOT) dataset and introduced seven types of common corruptions to its test set. These corruptions were categorized into weather (fog, rain, low-light), sensor noise (ISO noise, color quantization), and defocus. Each corruption was applied at four different severity levels to create a comprehensive evaluation suite.
  2. Extensive Experimental Evaluation: They evaluated eight diverse object detection models (including YOLOv5, YOLOv8, YOLOX, RetinaNet, Faster R-CNN, DiffusionDet, DETR, and CenterNet2) on both the original AOT dataset and their new AOT-C benchmark. This allowed them to analyze how different model architectures and types (one-stage vs. multi-stage, transformer-based) perform under various corruptions.
  3. Fine-tuning for Robustness: The researchers also explored the impact of using their synthetically generated corruptions as augmentation during the fine-tuning process. They then tested these fine-tuned models on real-world flight data to assess improvements in generalization.

Datasets

  • Airborne Object Tracking (AOT) Dataset: Used as the base dataset for creating the AOT-C benchmark.
  • AOT-C Dataset: A newly introduced benchmark dataset derived from AOT by applying seven types of common corruptions at four severity levels. This is the first robustness benchmark specifically for air-to-air aerial object detection.
  • Real-world Flight Tests: A custom dataset comprising 10,641 images from two different drones, used to validate the performance of models fine-tuned with synthetic corruptions.

Key Insights and Ideas

  • YOLO Family Shines: One-stage detectors, particularly those from the YOLO family (YOLOv5 and YOLOX), demonstrated superior robustness against common corruptions compared to other architectures. This is significant because they achieve this robustness without a substantial drop in performance on clean data.
  • Transformers are Fragile: Transformer-based and multi-stage detectors (like Faster R-CNN and DETR) were found to be highly vulnerable to corruptions, experiencing significant performance degradation.
  • Robustness = Generalization: The paper strongly suggests that a model's robustness against corruptions is directly linked to its ability to generalize to unseen data and challenging real-world scenarios. Models that perform well under corruption also tend to perform better in domain-shifted real-world flights.
  • Synthetic Data Augmentation Works: Fine-tuning models with the synthetically generated corruptions significantly improved their performance on real-world flight tests, demonstrating the effectiveness of this approach for enhancing generalization.

Key Results

  • Overall Robustness: The average drop in performance (APcor) across all models due to corruptions was 22.7% compared to clean data (APclean).
  • Best Performing Detector: YOLOv5 achieved the highest robustness against corruptions with an APcor of 51.4%.
  • Worst Performing Detector: RetinaNet showed the lowest robustness with an APcor of 20.0%.
  • Impact of Corruptions: ISO noise, near focus, and color quantization corruptions caused the most significant degradation in detection accuracy across all models. Adverse weather conditions like fog and rain had a less severe impact.
  • Fine-tuning Gains: Fine-tuned YOLOv5 showed an 11.3% improvement in AP on real-world flight tests compared to its base model.

Limitations and Future Work

  • Limited Corruption Set: While the AOT-C dataset covers seven common corruption types, the authors acknowledge that it is not exhaustive and encourage future work to include a wider range of corruptions.
  • Synthetic vs. Real-World Gap: Although synthetic corruptions showed promise, the paper notes that there's an inevitable gap between synthetic and real-world corruptions. Further research could focus on bridging this gap more effectively.
  • Model Architectures: The study evaluated eight detectors, but the field is rapidly evolving. Future work could extend this analysis to newer architectures.
  • Beyond RGB: The study focused on RGB monocular cameras. Investigating robustness with other sensor modalities (e.g., thermal, lidar) could be a valuable future direction.

Summary updated

Abstract

The main barrier to achieving fully autonomous flights lies in autonomous aircraft navigation. Managing non-cooperative traffic presents the most important challenge in this problem. The most efficient strategy for handling non-cooperative traffic is based on monocular video processing through deep learning models. This letter contributes to the vision-based deep learning aircraft detection and tracking literature by investigating the impact of data corruption arising from environmental and hardware conditions on the effectiveness of these methods. More specifically, we designed 7 types of common corruptions for camera inputs taking into account real-world flight conditions. By applying these corruptions to the Airborne Object Tracking (AOT) dataset we constructed the first robustness benchmark dataset named AOT-C for air-to-air aerial object detection. The corruptions included in this dataset cover a wide range of challenging conditions such as adverse weather and sensor noise. The second main contribution of this letter is to present an extensive experimental evaluation involving 8 diverse object detectors to explore the degradation in the performance under escalating levels of corruptions (domain shifts). Based on the evaluation results, the key observations that emerge are the following: 1) One-stage detectors of the YOLO family demonstrate better robustness, 2) Transformer-based and multi-stage detectors like Faster R-CNN are extremely vulnerable to corruptions, 3) Robustness against corruptions is related to the generalization ability of models. The third main contribution is to present that finetuning on our augmented synthetic data results in improvements in the generalisation ability of the object detector in real-world flight experiments.
BibTeX
@article{arsenos2024common,
  author = {Anastasios Arsenos and Vasileios Karampinis and Evangelos Petrongonas and Christos Skliros and Dimitrios Kollias and Stefanos D. Kollias and Athanasios Voulodimos},
  title = {{Common Corruptions for Evaluating and Enhancing Robustness in Air-to-Air Visual Object Detection}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  url = {https://openreview.net/forum?id=lJvftuCEkW},
}
Figure