Rapid Airplane Detection in Remote Sensing Images Based on Multilayer Feature Fusion in Fully Convolutional Neural Networks

Yuelei Xu, Mingming Zhu, Peng Xin, Shuai Li, Min Qi, Shiping MaSensors 2018 · DOI

TL;DR

This paper introduces a novel deep learning approach for rapid and accurate airplane detection in remote sensing images. By fusing features from different layers of a Convolutional Neural Network (CNN) and adapting the network architecture, they achieve significantly improved detection rates and reduced false alarms, particularly for small and cluttered airplane objects.

What Problem Does This Paper Address?

Detecting airplanes in aerial and satellite imagery, especially those on the ground, presents several challenges. Airplanes can appear as small objects with varying scales, colors, and orientations, often against complex backgrounds. Traditional methods struggle with low accuracy, high false alarm rates, and slow processing speeds, making them unsuitable for real-time applications like reconnaissance or status monitoring. Existing deep learning approaches, while better, can still suffer from issues like low dimensionality in deep features and inadequate representation of small objects, leading to missed detections or misclassifications.

Methods

The core of this paper's contribution lies in a modified Fully Convolutional Neural Network (FCNN) architecture designed for airplane detection. Key methodological aspects include:

  • Multilayer Feature Fusion: Instead of relying solely on the output of the deepest layers (which have high semantic but low spatial information), the authors fuse features from shallow and deep layers of the CNN. To achieve this, shallow layer features are down-sampled, and deep layer features are up-sampled to a common scale before fusion. This aims to combine the precise positional information from shallow layers with the abstract semantic information from deep layers.
  • Architecture Modifications:
    • Replacing Fully Connected Layers: Fully connected layers are replaced with convolutional layers. This reduces the number of network parameters and allows the network to handle input images of varying sizes more effectively.
  • Region Proposal Network (RPN) Integration: The Region Proposal Network (RPN), responsible for identifying potential airplane locations, shares convolutional layers with the detection network. This significantly boosts detection efficiency by avoiding redundant computations.
  • Adaptive Candidate Region Sizing: The sizes and aspect ratios of candidate regions generated by the RPN are specifically tailored to better match the typical characteristics of airplanes in aerial images.
  • Training Strategy: A multi-stage training process is employed, involving an RPN and a detection network. This strategy ensures that the convolutional layers are shared between the two networks, optimizing for efficiency. A multitask loss function is used to jointly train classification and bounding box regression.

Datasets

The study utilized satellite images collected from 200 airports worldwide, including locations like Ronald Reagan Washington National Airport and Beijing Capital International Airport. The images were captured in December 2017, with resolutions ranging from 1 to 10 meters and sizes between 800x600 and 1200x800 pixels.

  • Data Splitting: 50% of the images were used for training and validation, with the remaining 50% forming the test dataset.
  • Data Augmentation: To address airplane rotation and prevent overfitting, training images were augmented by rotating them by 90°, 180°, and 270°, and by flipping them horizontally.
  • Total Training Data: After augmentation, the training dataset comprised approximately 250 training data points.
  • Annotation: Only airplane objects were labeled with bounding boxes for training and testing.

Key Insights and Ideas

  • The Power of Fusion: Fusing features from different CNN layers is crucial for robust airplane detection. Relying on only the deepest layer's features leads to poor performance due to the loss of spatial information, especially for small objects.
  • Balancing Semantics and Position: Shallow layers provide precise location information, while deep layers offer rich semantic understanding. Combining these two is key to accurately localizing and identifying airplanes.
  • Architectural Efficiency: Replacing fully connected layers with convolutional layers and sharing layers between the RPN and detection network are effective strategies for improving speed and adaptability to different input sizes.
  • Tailoring to the Object: Adjusting the candidate region generation parameters to match the typical shape and size of airplanes significantly improves detection.

Key Results

  • Improved Detection Rate: The proposed method achieved a detection rate of 95.5%, outperforming comparative methods like Location-DBN (83.5%), BING-CNN (85.4%), Fast RCNN (81.2%), and Faster RCNN (82.7%).
  • Reduced False Alarms: The false alarm rate was significantly lower at 7.5% compared to other methods (Location-DBN: 36.5%, BING-CNN: 20.4%, Fast RCNN: 22.5%, Faster RCNN: 23.5%).
  • Faster Detection: The average detection time was 0.3 seconds, which is competitive and efficient, especially considering the improvements in accuracy and false alarm reduction.
  • Effective for Small Objects: Qualitative results demonstrated the method's capability to accurately detect small airplane objects, which is a common challenge.

Limitations and Future Work

  • Occasional Failures: The authors acknowledge that some detection failures can occur. These include misidentifying objects similar to airplanes (Figure 7a) and failing to detect airplanes in highly cluttered scenes where bounding boxes overlap significantly (Figure 7b).
  • Need for Improvement: Despite significant improvements, the method still requires further refinement.
  • Future Direction: The authors plan to focus future research on incorporating contextual information into the detection process. This suggests that understanding the surrounding environment of an airplane could further enhance detection accuracy and reduce false positives.

Summary updated

Abstract

To address the issues encountered when using traditional airplane detection methods, including the low accuracy rate, high false alarm rate, and low detection speed due to small object sizes in aerial remote sensing images, we propose a remote sensing image airplane detection method that uses multilayer feature fusion in fully convolutional neural networks. The shallow layer and deep layer features are fused at the same scale after sampling to overcome the problems of low dimensionality in the deep layer and the inadequate expression of small objects. The sizes of candidate regions are modified to fit the size of the actual airplanes in the remote sensing images. The fully connected layers are replaced with convolutional layers to reduce the network parameters and adapt to different input image sizes. The region proposal network shares convolutional layers with the detection network, which ensures high detection efficiency. The simulation results indicate that, when compared to typical airplane detection methods, the proposed method is more accurate and has a lower false alarm rate. Additionally, the detection speed is considerably faster and the method can accurately and rapidly complete airplane detection tasks in aerial remote sensing images.
BibTeX
@inproceedings{xu2018rapid,
  author = {Yuelei Xu and Mingming Zhu and Peng Xin and Shuai Li and Min Qi and Shiping Ma},
  title = {{Rapid Airplane Detection in Remote Sensing Images Based on Multilayer Feature Fusion in Fully Convolutional Neural Networks}},
  booktitle = {Sensors},
  year = {2018},
  doi = {10.3390/s18072335},
}
Figure