Object Detection for UAV Aerial Scenarios Based on Vectorized IOU

Shun Lu, Hanyu Lu, Jun Dong, Shuang WuSensors 2023 · DOI

TL;DR

This paper tackles the challenging problem of object detection in UAV aerial imagery, which is plagued by small, overlapping, and multi-scale objects. They introduce a novel Vectorized Intersection Over Union (VIOU) loss for more accurate bounding box regression, a Progressive Feature Fusion Network (PFFN) to better integrate shallow and deep features, and an Asymmetric Decoupled (AD) head to improve classification and localization. These improvements lead to significant performance gains on benchmark datasets.

What Problem Does This Paper Address?

Detecting objects in images captured by Unmanned Aerial Vehicles (UAVs) is a notoriously difficult task. Unlike typical ground-level imagery, UAV aerial images present unique challenges:

  • Multi-scale Objects: Objects can appear at vastly different sizes depending on their altitude and distance from the camera.
  • High Proportion of Small Objects: Many objects of interest (e.g., pedestrians, vehicles) are small in the image frame.
  • High Overlap: Objects can be densely packed and heavily overlap, making it hard to distinguish individual instances.
  • Limited Semantic Information in Shallow Features: Standard convolutional neural networks (CNNs) often lose fine-grained details in early layers as they downsample, which are crucial for detecting small objects. Conversely, deeper layers have rich semantic information but low resolution.

Existing object detection methods, even advanced ones like YOLOv5, struggle to effectively handle these issues, leading to missed detections and inaccurate bounding boxes.

Methods

The authors propose a three-pronged approach to enhance object detection in UAV aerial scenarios, building upon the YOLOv5s architecture:

  1. Vectorized Intersection Over Union (VIOU) Loss: This novel loss function aims to improve bounding box regression accuracy. Instead of just considering the overlap (IOU) and aspect ratio, VIOU treats the width and height of bounding boxes as vectors. It uses a cosine function to directly compare the center point and the vector representations of width and height, aiming for more direct and accurate regression of these parameters. This is particularly beneficial for small objects where minor errors have a larger impact.
  2. Progressive Feature Fusion Network (PFFN): To address the limitations of shallow feature extraction, the authors introduce PFFN. This network enhances the feature fusion process by allowing each node to integrate semantic information from deeper layers with features from its own layer. This "progressive" fusion, particularly the "version 4" approach (increasing fusion from top to bottom), ensures that detailed features from shallow layers are preserved while benefiting from the rich semantic context of deeper layers. They also add a dedicated detection head for small objects at a higher resolution (160x160).
  3. Asymmetric Decoupled (AD) Head: Recognizing that classification and regression are distinct tasks with different requirements, the authors propose an AD head. This head decouples these two tasks into separate branches. The classification branch uses convolutional kernels of varying sizes (1x1, 3x3, 5x5) to capture multi-scale contextual information for better category identification. The regression branch focuses on refining bounding box coordinates, width, and height, aiming for more precise localization.

Datasets

The proposed method was evaluated on two widely used benchmark datasets for aerial object detection:

  • VisDrone 2019-DET: This dataset is specifically designed for UAV-based object detection and tracking. It contains a large number of small objects and challenging scenarios like high overlap.
  • DOTA: A large-scale dataset for object detection in aerial images, featuring 15 categories of remote-sensing objects.

Key Insights and Ideas

  • Vectorizing Bounding Box Attributes: Treating bounding box width and height as vectors and using their norms and angles in a loss function offers a more nuanced way to guide regression, especially for small objects where aspect ratio and size are critical.
  • Bridging the Semantic-Detail Gap: The PFFN's progressive fusion strategy effectively combines the strengths of shallow (detailed, high-resolution) and deep (semantic, low-resolution) features, a common challenge in CNNs. This is crucial for detecting objects of varying scales.
  • Task Decoupling for Specialization: Separating classification and regression into specialized branches (AD head) allows each task to optimize its own objectives more effectively, leading to improved overall performance. The different receptive fields in the classification branch and the focused refinement in the regression branch highlight this specialization.
  • UAV Detection is Still a Hard Problem: The paper's comparison with existing state-of-the-art detectors on VisDrone 2019 shows that even advanced methods struggle, underscoring the need for specialized approaches.

Key Results

The proposed method, integrating VIOU loss, PFFN, and AD head, demonstrated significant improvements over the baseline YOLOv5s:

  • VisDrone 2019 Dataset: Achieved an [email protected] of 44.6%, a 9.7% increase from the baseline's 34.9%. This performance also surpassed other state-of-the-art detectors.
  • DOTA Dataset: Achieved a 2.1% performance improvement over YOLOv5s.
  • VIOU Loss Parameter Tuning: The optimal value for the parameter 'a' in the VIOU loss was found to be 1 for the VisDrone dataset, yielding an [email protected] of 36.4%.
  • Ablation Studies: Each component (VIOU, PFFN, AD head) contributed to performance gains, with the full model showing the best results. The PFFN, in particular, significantly improved all evaluation metrics.

Limitations and Future Work

  • Computational Cost: While the paper focuses on improving accuracy, the addition of PFFN and AD head might increase computational complexity and inference time. The paper notes that the AD head reduces calculations compared to the classification branch, but overall inference time for the full model is higher than the baseline (7.1 ms vs. 1.5 ms on VisDrone).
  • Dataset Specificity: The optimal VIOU parameter 'a' was determined for the VisDrone dataset. Further investigation might be needed to generalize this or find optimal parameters for other datasets.
  • Future Work: The authors suggest that their methods are easily implementable and convenient for practical scenarios, implying potential for broader application. The paper doesn't explicitly detail future research directions, but the modular nature of their contributions (loss, neck, head) suggests that these components could be integrated into other object detection frameworks.

Summary updated

Abstract

Object detection in unmanned aerial vehicle (UAV) images is an extremely challenging task and involves problems such as multi-scale objects, a high proportion of small objects, and high overlap between objects. To address these issues, first, we design a Vectorized Intersection Over Union (VIOU) loss based on YOLOv5s. This loss uses the width and height of the bounding box as a vector to construct a cosine function that corresponds to the size of the box and the aspect ratio and directly compares the center point value of the box to improve the accuracy of the bounding box regression. Second, we propose a Progressive Feature Fusion Network (PFFN) that addresses the issue of insufficient semantic extraction of shallow features by Panet. This allows each node of the network to fuse semantic information from deep layers with features from the current layer, thus significantly improving the detection ability of small objects in multi-scale scenes. Finally, we propose an Asymmetric Decoupled (AD) head, which separates the classification network from the regression network and improves the classification and regression capabilities of the network. Our proposed method results in significant improvements on two benchmark datasets compared to YOLOv5s. On the VisDrone 2019 dataset, the performance increased by 9.7% from 34.9% to 44.6%, and on the DOTA dataset, the performance increased by 2.1%.
BibTeX
@inproceedings{lu2023object,
  author = {Shun Lu and Hanyu Lu and Jun Dong and Shuang Wu},
  title = {{Object Detection for UAV Aerial Scenarios Based on Vectorized IOU}},
  booktitle = {Sensors},
  year = {2023},
  doi = {10.3390/s23063061},
}
Figure