SRE-YOLOv8: An Improved UAV Object Detection Model Utilizing Swin Transformer and RE-FPN

Jun Li, Jiajie Zhang, Yanhua Shao, Feng Liu2024 · DOI

TL;DR

This paper introduces SRE-YOLOv8, an enhanced object detection model for Unmanned Aerial Vehicles (UAVs) that significantly improves accuracy, especially for small objects. By integrating the Swin Transformer for better feature extraction and a novel RE-FPN for improved feature fusion, the model achieves a remarkable 9.2% increase in mAP on the VisDrone2021 dataset.

What Problem Does This Paper Address?

Detecting objects in aerial imagery from UAVs presents unique challenges. UAVs capture images from high altitudes with wide-angle lenses, resulting in objects appearing small, with weak features, and often densely packed or in complex backgrounds. Traditional object detection methods struggle with these issues, leading to low accuracy. Existing models like YOLOv8, while powerful, can still miss small objects and lack robust contextual understanding in challenging aerial scenes. This paper aims to bridge this gap by developing a more robust and accurate object detection system specifically for UAV imagery.

Methods

The SRE-YOLOv8 model builds upon the YOLOv8 architecture with several key enhancements:

  • Enhanced Backbone (C2f-ST): The standard CSPDarknet53 backbone is augmented by integrating an improved Swin Transformer module (C2f-ST). This leverages the Swin Transformer's ability to capture global contextual information through its window-based self-attention mechanism, which is crucial for understanding complex aerial scenes.
  • Lightweight Residual Feature Pyramid Network (RE-FPN): The original FPN is replaced with a novel RE-FPN. This structure incorporates a Residual Feature Augmentation (RFA) module to inject contextual information from different spatial locations and an Efficient Channel Attention (ECA) module to focus on critical features without significantly increasing computational load. This aims to improve feature representation and mitigate spatial information loss.
  • Small Object Detection (SOD) Layer: An additional SOD layer is introduced in the neck of the network. This layer fuses features from different scales and utilizes larger-scale feature maps to enhance the model's ability to detect extremely small objects by better capturing their spatial details.
  • Dynamic Head: The detection head is enhanced with a Dynamic Head architecture. This employs multiple attention mechanisms (scale-aware, spatial-aware, and task-aware) to dynamically adjust the model's focus, improving its expressive capabilities for target representation and enhancing detection accuracy, especially for small and low-resolution targets in complex backgrounds.

Datasets

The primary dataset used for training and evaluation in this paper is VisDrone2021. This dataset is specifically designed for UAV object detection and contains images captured from various locations and environments, featuring a wide range of object sizes and densities, including a significant number of small objects. The dataset comprises 7019 images for training and 548 for validation, with an additional 1610 images for testing.

Key Insights and Ideas

  • Swin Transformer for Global Context: The integration of Swin Transformer into the backbone is a critical step. It highlights how leveraging self-attention mechanisms can significantly improve the model's understanding of the broader scene, which is often lost in traditional CNNs when dealing with aerial imagery.
  • RE-FPN for Efficient Feature Fusion: The development of RE-FPN demonstrates a clever approach to improving feature fusion. By combining residual connections with lightweight attention, the authors show it's possible to enhance feature representation without the computational overhead of more complex structures like BiFPN.
  • Dedicated SOD Layer is Crucial: The explicit addition of an SOD layer underscores the persistent challenge of small object detection. It suggests that specialized modules are still necessary to effectively handle objects that occupy very few pixels.
  • Dynamic Head for Adaptive Focus: The Dynamic Head's multi-dimensional attention mechanism is a sophisticated way to make the detection head more adaptive. It shows that by dynamically adjusting focus based on scale, space, and task, the model can better handle the diverse and challenging conditions in UAV imagery.

Key Results

The SRE-YOLOv8 model demonstrated significant improvements over the baseline YOLOv8:

  • Overall Accuracy: A substantial 9.2% increase in mean average precision ([email protected]) was achieved.
  • Category-Specific Improvements: Approximately 10% improvements in AP were observed for categories like pedestrian, car, tricycle, awning-tricycle, and motor.
  • Qualitative Improvements: Representative detection results show the SRE-YOLOv8 model is better at detecting more small targets and handling dense, low-resolution objects in UAV aerial images compared to the original YOLOv8.
  • Ablation Study: The ablation study confirmed the effectiveness of each proposed module. The C2f-ST module alone provided a notable 5.8% increase in mAP, while the combination of all modules led to the highest performance gains.

Limitations and Future Work

  • Inference Speed: While the SRE-YOLOv8 model achieves higher accuracy, the paper notes that some enhancements, particularly the C2f-ST module, slightly increase training load and can lead to a decrease in inference speed. The authors suggest this can be mitigated by improving hardware performance.
  • Specific Categories: The paper acknowledges that for certain categories (e.g., tricycle, people, awning-tricycle), the FN (false negative) values are relatively high. This is attributed to the lower number of training samples for these categories, leading to limited feature extraction.
  • Future Directions: The authors intend to explore methods for further enhancing small object detection accuracy while simultaneously mitigating computational complexity and accelerating inference. This suggests a continued focus on efficiency and robustness for drone aerial scenarios with constrained computing resources.

Summary updated

BibTeX
@misc{li2024sre,
  author = {Jun Li and Jiajie Zhang and Yanhua Shao and Feng Liu},
  title = {{SRE-YOLOv8: An Improved UAV Object Detection Model Utilizing Swin Transformer and RE-FPN}},
  year = {2024},
  doi = {10.3390/s24123918},
}
Figure