SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery

Jiaqing Zhang, Jie Lei, Weiying Xie, Zhenman Fang, Yunsong Li, Qian Du

TL;DR

SuperYOLO introduces a lightweight, real-time object detection framework for multimodal remote sensing imagery that achieves state-of-the-art accuracy on small objects without the typical computational overhead. By leveraging pixel-level fusion and an auxiliary super-resolution branch that is discarded during inference, it delivers a massive accuracy boost while remaining highly efficient for on-board deployment.

What Problem Does This Paper Address?

Detecting objects in remote sensing imagery (RSI)—such as drones or satellite photos—is notoriously difficult. The targets (like cars or boats) are often tiny, spanning only a few dozen pixels against massive, complex backgrounds. Standard deep learning models tend to lose these small objects because their architectures aggressively downsample images to save on computation.

While utilizing multiple sensor modalities (such as combining standard RGB images with Infrared) can provide helpful complementary data, existing fusion networks are computationally heavy. This makes them impractical for real-time, on-board processing on edge devices. The authors of this paper tackle this exact gap: how to achieve high-resolution detection accuracy for tiny objects without introducing a massive computational burden.

Methods

The authors build their framework on top of the compact and fast YOLOv5s architecture, introducing three key innovations to create SuperYOLO:

  1. Focus Module Removal: Standard YOLOv5 uses a "Focus" module to slice and downsample images early on to accelerate training. SuperYOLO removes this module entirely to preserve high-resolution spatial details that are critical for identifying small targets.
  2. Symmetric Pixel-Level Multimodal Fusion (MF): Instead of using expensive parallel networks to extract features from RGB and Infrared (IR) data separately, SuperYOLO fuses them at the pixel level. It employs Squeeze-and-Excitation (SE) blocks to efficiently extract inner relationships across both spatial and channel domains.
  3. Assisted Super-Resolution (SR) Branch: To force the backbone network to retain fine-grained spatial details, the authors attach an auxiliary SR branch during training. This branch learns to reconstruct high-resolution images from the backbone's intermediate features. The clever twist: this entire SR branch is completely discarded during the inference (testing) stage.
Overview of the proposed SuperYOLO framework

Datasets

The model was trained and evaluated on several prominent remote sensing benchmarks:

  • VEDAI: The primary dataset used for evaluation. It features co-registered RGB and Infrared aerial images focusing on multi-class vehicles against diverse backgrounds.
  • Single-Modality Datasets (DOTA, NWPU VHR-10, DIOR): To validate the model's generalization capabilities, the authors also tested SuperYOLO on these large-scale, standard optical remote sensing datasets.

Key Insights and Ideas

  • The "Free" Training Signal: The auxiliary SR branch acts as a regularizer during training, guiding the backbone network to learn high-resolution feature representations. Because it is discarded during deployment, it provides a significant accuracy boost with absolutely zero computational penalty at inference time.
  • The Focus Module Trade-off: While removing the Focus module increases GFLOPs (computational complexity) slightly, the spatial information retained is so vital for small objects that it yields a massive accuracy jump (>5% mAP) across all YOLOv5 variants.
  • Pixel-Level Over Feature-Level Fusion: The authors experimented with fusing data at different stages of the network. They discovered that pixel-level fusion avoids the accumulation of redundant information and keeps the parameter count low, making it much more effective for real-time applications than complex feature-level alternatives.
Architecture of the pixel-level multimodal fusion module

Key Results

  • State-of-the-Art Accuracy: On the VEDAI dataset, SuperYOLO achieved an accuracy of 75.09% mAP50, outperforming much larger models like YOLOv5x by over 10%.
  • Highly Efficient: SuperYOLO's parameter size is about 18x smaller and its GFLOPs are 3.8x less than YOLOv5x, demonstrating an exceptional accuracy-speed trade-off.
  • Excellent Generalization: When tested on single-modality datasets (DOTA, NWPU, DIOR), SuperYOLO consistently achieved optimal detection results compared to other SOTA detectors, proving its architectural modifications are universally beneficial for remote sensing tasks.

Limitations and Future Work

Although SuperYOLO successfully reduces inference-time computation by discarding the SR branch, the authors note that the network still relies heavily on high-resolution feature retention in the backbone. For future work, they plan to focus on designing even lower-parameter feature extraction modules to further optimize the framework for ultra-low-power, real-time on-board processing.

Summary updated

Abstract

Accurately and timely detecting multiscale small objects that contain tens of pixels from remote sensing images (RSI) remains challenging. Most of the existing solutions primarily design complex deep neural networks to learn strong feature representations for objects separated from the background, which often results in a heavy computation burden. In this article, we propose an accurate yet fast object detection method for RSI, named SuperYOLO, which fuses multimodal data and performs high-resolution (HR) object detection on multiscale objects by utilizing the assisted super resolution (SR) learning and considering both the detection accuracy and computation cost. First, we utilize a symmetric compact multimodal fusion (MF) to extract supplementary information from various data for improving small object detection in RSI. Furthermore, we design a simple and flexible SR branch to learn HR feature representations that can discriminate small objects from vast backgrounds with low-resolution (LR) input, thus further improving the detection accuracy. Moreover, to avoid introducing additional computation, the SR branch is discarded in the inference stage, and the computation of the network model is reduced due to the LR input. Experimental results show that, on the widely used VEDAI RS dataset, SuperYOLO achieves an accuracy of 75.09% (in terms of mAP50 ), which is more than 10% higher than the SOTA large models, such as YOLOv5l, YOLOv5x, and RS designed YOLOrs. Meanwhile, the parameter size and GFLOPs of SuperYOLO are about 18 times and 3.8 times less than YOLOv5x. Our proposed model shows a favorable accuracy and speed tradeoff compared to the state-of-the-art models. The code will be open-sourced at https://github.com/icey-zhang/SuperYOLO.
BibTeX
@article{zhang2023superyolo,
  author = {Jiaqing Zhang and Jie Lei and Weiying Xie and Zhenman Fang and Yunsong Li and Qian Du},
  title = {{SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2023},
  doi = {10.1109/tgrs.2023.3258666},
}
Figure