Ensuring UAV Safety: A Vision-only and Real-time Framework for Collision Avoidance Through Object Detection, Tracking, and Distance Estimation

Vasileios Karampinis, Anastasios Arsenos, Orfeas Filippopoulos, Evangelos Petrongonas, Christos Skliros, Dimitrios Kollias, Stefanos D. Kollias, Athanasios VoulodimosarXiv 2024 · OpenReview

TL;DR

This paper introduces a novel vision-only, real-time framework for UAVs to detect, track, and estimate the distance of other aircraft, crucial for preventing mid-air collisions. By treating depth estimation as an image-to-image translation task with a lightweight encoder-decoder network, they achieve robust performance on the challenging Airborne Object Tracking (AOT) dataset.

What Problem Does This Paper Address?

Mid-air collisions (MACs) and near mid-air collisions (NMACs) pose significant safety risks for both manned and unmanned aircraft, especially in low-altitude airspace. While cooperative aircraft can rely on transponders for collision avoidance, a substantial threat comes from non-cooperative aircraft (e.g., rogue drones, gliders) that lack such systems. Existing sense-and-avoid systems often rely on radar, which can be too bulky for small UAVs. Therefore, there's a critical need for lightweight, vision-based solutions that can accurately detect, track, and estimate the distance to these non-cooperative airborne threats in real-time, enabling autonomous collision avoidance.

Methods

The proposed framework integrates three key deep learning components: object detection, tracking, and distance estimation.

  • Object Detection & Tracking: The system leverages a pre-existing pipeline (NEFELI [13]) for detecting aircraft. This detection is then fed into a tracking module. High-confidence detections are managed by an appearance-based Re-ID model, while low-confidence ones use a Kalman Filter (KF) with camera motion compensation (CMC) to maintain track continuity.
  • Distance Estimation: This is the core novelty. Instead of integrating depth estimation directly into the object detector, the authors frame it as an image-to-image translation problem. They employ a lightweight encoder-decoder network (U-Net architecture) for this task. This approach allows for efficient and robust depth estimation from a monocular camera. The encoder extracts hierarchical features, and the decoder upsamples them to produce a depth map.

Datasets

  • Training: The Amazon Airborne Object Tracking (AOT) Dataset was used. The authors ingeniously constructed ground truth depth maps for training by leveraging the provided bounding box annotations and GPS-derived distance information from the AOT dataset. This involved creating a classification scheme for depth ranges.
  • Evaluation: The Airborne Object Tracking (AOT) Dataset was also used for evaluating the performance of the entire pipeline.

Key Insights and Ideas

  • Depth Estimation as Image-to-Image Translation: Framing depth estimation as an image-to-image translation task, rather than a regression problem directly integrated with object detection, allows for a more specialized and potentially more accurate depth estimation module.
  • Lightweight Encoder-Decoder for Depth: Utilizing a lightweight encoder-decoder network is crucial for achieving real-time performance on resource-constrained UAVs, avoiding the need for high-end GPUs.
  • Hybrid Loss Function for Depth: Combining multiple loss functions (Edge, SSIM, L1, and Berhu) proved more effective than using individual losses for depth estimation, leading to better performance and robustness. The Berhu loss, in particular, showed promise for its ability to handle outliers.
  • Classification-based Depth: Treating depth estimation as a classification problem into discrete bins (e.g., <200m, <400m) rather than a continuous regression task can be more suitable for collision avoidance thresholds.

Key Results

  • The proposed framework achieved remarkable accuracy on the AOT dataset for object detection, tracking, and distance estimation.
  • The multi-loss trained model (combining Edge, SSIM, L1, and Berhu losses) demonstrated superior performance in classification metrics compared to models trained with individual losses.
  • Specifically, the mean function in the sliding window approach for deriving final depth predictions outperformed max and min functions, highlighting its robustness against outliers. The min function also showed strong performance.
  • The Edge loss alone resulted in poor performance, underscoring its role as a component within a larger loss strategy rather than a standalone solution for depth estimation.
  • The L1 loss function was capable of pinpointing the area of interest but struggled with defining outliers effectively.

Limitations and Future Work

  • Dataset Assumptions: The authors acknowledge that their method of creating depth ground truths from the AOT dataset involves assumptions, such as the bounding box strictly enclosing the object of interest, which might not perfectly reflect real-world scenarios. However, they note that the bias from these assumptions was negligible in their evaluation.
  • Domain Shifts: The models are trained on a benchmark dataset (AOT). A significant future direction is to test the models' robustness against domain shifts caused by various weather conditions (rain, fog, clouds) and hardware deficiencies.
  • Real-World Flight Tests: Exposing the models to authentic flight tests is paramount to assessing their adaptability to adverse weather and unforeseen real-world constraints, which are difficult to replicate in supervised environments.
  • Deployment Challenges: Deploying a model trained on a benchmark dataset to a realistic environment is inherently challenging due to the potential for extensive data discrepancies.

Summary updated

Abstract

In the last twenty years, unmanned aerial vehicles (UAVs) have garnered growing interest due to their expanding applications in both military and civilian domains. Detecting non-cooperative aerial vehicles with efficiency and estimating collisions accurately are pivotal for achieving fully autonomous aircraft and facilitating Advanced Air Mobility (AAM). This paper presents a deep-learning framework that utilizes optical sensors for the detection, tracking, and distance estimation of non-cooperative aerial vehicles. In implementing this comprehensive sensing framework, the availability of depth information is essential for enabling autonomous aerial vehicles to perceive and navigate around obstacles. In this work, we propose a method for estimating the distance information of a detected aerial object in real time using only the input of a monocular camera. In order to train our deep learning components for the object detection, tracking and depth estimation tasks we utilize the Amazon Airborne Object Tracking (AOT) Dataset. In contrast to previous approaches that integrate the depth estimation module into the object detector, our method formulates the problem as image-to-image translation. We employ a separate lightweight encoder-decoder network for efficient and robust depth estimation. In a nutshell, the object detection module identifies and localizes obstacles, conveying this information to both the tracking module for monitoring obstacle movement and the depth estimation module for calculating distances. Our approach is evaluated on the Airborne Object Tracking (AOT) dataset which is the largest (to the best of our knowledge) air-to-air airborne object dataset.
BibTeX
@misc{karampinis2024ensuring,
  author = {Vasileios Karampinis and Anastasios Arsenos and Orfeas Filippopoulos and Evangelos Petrongonas and Christos Skliros and Dimitrios Kollias and Stefanos D. Kollias and Athanasios Voulodimos},
  title = {{Ensuring UAV Safety: A Vision-only and Real-time Framework for Collision Avoidance Through Object Detection, Tracking, and Distance Estimation}},
  year = {2024},
  eprint = {2405.06749},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.06749},
}
Figure