Effective Airplane Detection in Remote Sensing Images Based on Multilayer Feature Fusion and Improved Nonmaximal Suppression Algorithm

Mingming Zhu, Yuelei Xu, Shiping Ma, Shuai Li, Hongqiang Ma, Yongsai HanRemote 2019 · DOI

TL;DR

This paper introduces a novel deep learning approach for detecting airplanes in remote sensing images. It tackles the challenges of small, weak objects and overlapping detections by fusing features from multiple network layers and employing a refined non-maximum suppression technique, achieving superior accuracy and efficiency.

What Problem Does This Paper Address?

Detecting airplanes in remote sensing imagery is a critical task for applications like surveillance and aviation management. However, it's a challenging problem due to several factors: airplanes can appear as small, weak objects against complex backgrounds, and multiple detection boxes can overlap, leading to false positives or missed detections. Traditional methods often struggle with these issues, and while deep learning has shown promise, adapting it to the unique characteristics of remote sensing data, especially with limited labeled datasets, remains an active research area.

Methods

The proposed method builds upon the Faster R-CNN framework, a popular region-based convolutional neural network architecture for object detection. The core innovations are:

  • Multilayer Feature Fusion: To better represent small and weak objects, the authors fuse features extracted from different convolutional layers of the network. This is achieved through a sequence of L2 norm normalization, feature concatenation, scale scaling, and dimension reduction. This process aims to combine the spatial detail from lower layers with the semantic richness of higher layers.
  • Improved Non-Maximum Suppression (NMS): A common issue in object detection is the generation of multiple bounding boxes for the same object. Traditional NMS uses a hard threshold to eliminate overlapping boxes. This paper proposes a "soft decision" NMS that attenuates the scores of overlapping boxes rather than outright discarding them. This approach is designed to reduce missed detections, especially in cases of high object overlap.
  • Transfer Learning: Given the scarcity of labeled remote sensing data, the authors leverage transfer learning. They initialize their network with weights pre-trained on a large natural image dataset (VGG16) and then fine-tune it on their airplane remote sensing dataset. This allows the model to benefit from general visual features learned from natural images.

Datasets

The study utilizes a custom remote sensing image dataset for airplane detection.

  • Training Data: Initially, 1500 remote sensing airport images were collected from sources like Google Earth. To augment this dataset and learn rotation-invariant features, these images were expanded through data augmentation (rotation, flipping) to 5000 images. 2500 were used for training.
  • Verification Data: 1500 images were set aside for verification.
  • Test Data: A separate set of 150 images, also manually labeled, was used for final evaluation. These images were of varying sizes and resolutions.

Key Insights and Ideas

  • Feature Fusion is Crucial for Small Objects: The paper strongly suggests that combining features from multiple layers (specifically layers 3, 4, and 5) is essential for effectively detecting small airplanes, which can become significantly downscaled after deep convolutional processing.
  • Soft NMS Prevents Missed Detections: The "soft decision" NMS is a key improvement. By attenuating scores instead of hard-thresholding, it significantly reduces the problem of missing airplanes that are close to each other or partially occluded, a common pitfall of traditional NMS.
  • Transfer Learning Bridges the Data Gap: The success of transfer learning highlights its importance in remote sensing applications where large, labeled datasets are expensive and time-consuming to create. Leveraging pre-trained models on natural images provides a strong foundation.

Key Results

The proposed method demonstrated superior performance compared to several existing airplane detection techniques:

  • Accuracy: Achieved an accuracy (AC) of 94.25%, which was higher than DBN-based (79.54%), BING+CNN (84.25%), Faster R-CNN (86.28%), and YOLOv2 (90.05%).
  • Error Reduction: Significantly reduced false positive rate (FPR) to 5.59% and missing ratio (MR) to 5.75%, leading to a lower overall error ratio (ER) of 11.34%.
  • Efficiency: The method was comparable in speed to Faster R-CNN and YOLOv2, with an average processing time of 0.16 seconds. While not real-time in the strictest sense, it offers a good balance of speed and accuracy.
  • Qualitative Improvements: Visual results showed that the method could detect more small objects and handle overlapping detections more effectively than traditional NMS.

Limitations and Future Work

  • Real-time Processing: While efficient, the method was not as fast as YOLOv2, indicating that achieving true real-time performance for complex remote sensing tasks remains an ongoing challenge.
  • Model Complexity vs. Efficiency: The authors acknowledge that increasing the number of layers in deep learning models can lead to more parameters and longer training times. Future work will focus on optimizing network structures to balance performance and efficiency.
  • Generalization: While the paper shows application to airport detection, further exploration of its generalization to other object detection problems in remote sensing is suggested.

Summary updated

Abstract

Aiming at the problem of insufficient representation ability of weak and small objects and overlapping detection boxes in airplane object detection, an effective airplane detection method in remote sensing images based on multilayer feature fusion and an improved nonmaximal suppression algorithm is proposed. Firstly, based on the common low-level visual features of natural images and airport remote sensing images, region-based convolutional neural networks are chosen to conduct transfer learning for airplane images using a limited amount of data. Then, the L2 norm normalization, feature connection, scale scaling, and feature dimension reduction are introduced to achieve effective fusion of low- and high-level features. Finally, a nonmaximal suppression method based on a soft decision function is proposed to solve the overlap problem of detection boxes. The experimental results show that the proposed method can effectively improve the representation ability of weak and small objects, as well as quickly and accurately detect airplane objects in the airport area.
BibTeX
@inproceedings{zhu2019effective,
  author = {Mingming Zhu and Yuelei Xu and Shiping Ma and Shuai Li and Hongqiang Ma and Yongsai Han},
  title = {{Effective Airplane Detection in Remote Sensing Images Based on Multilayer Feature Fusion and Improved Nonmaximal Suppression Algorithm}},
  booktitle = {Remote},
  year = {2019},
  doi = {10.3390/rs11091062},
}
Figure