SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery

Jiaqing Zhang, Jie Lei, Weiying Xie, Zhenman Fang, Yunsong Li, Qian Du

Introduction

Detecting objects in remote sensing imagery (RSI) presents a significant challenge due to the "small object" problem. Targets like vehicles or tractors often occupy only a few dozen pixels within vast, complex backgrounds. Standard object detectors, such as the YOLO family, typically downsample images early in the network to reduce computational load. However, this "resolution tax" often destroys the fine-grained features necessary to identify tiny targets.

Furthermore, relying on a single modality (like RGB) is unreliable in remote sensing due to environmental factors like weather, shadows, or poor lighting. While Infrared (IR) data offers a complementary perspective, fusing these disparate data types efficiently without significantly increasing the model's computational footprint remains a hurdle.

Methodology: The SuperYOLO Framework

SuperYOLO is a lightweight, real-time object detection framework built upon the YOLOv5s architecture, incorporating three critical modifications to address these challenges.

1. Preserving Spatial Resolution

Standard YOLOv5 utilizes a "Focus" module that partitions pixels to reduce computation. In SuperYOLO, this module is removed. The authors found that preserving high-resolution spatial information from the earliest stages of the network is vital for detecting small objects.

2. Symmetric Multimodal Fusion (MF)

Instead of complex feature-level fusion deep within the network, SuperYOLO performs fusion at the pixel level. It employs Squeeze-and-Excitation (SE) blocks to extract channel-wise information and $1 \times 1$ convolutions to bi-directionally fuse RGB and IR data. This compact approach provides a richer input to the backbone without a massive increase in GFLOPs.

3. Assisted Super Resolution (SR) Branch

The core innovation of SuperYOLO is the auxiliary Super Resolution branch used during training. This branch attempts to reconstruct a high-resolution version of the input image from the backbone's features. By penalizing the model (via L1 loss) for reconstruction errors, the network is forced to learn and retain fine-grained spatial details. Crucially, this branch is discarded during inference, allowing the model to benefit from high-resolution "thinking" during training without any speed penalty during deployment.

Comparison of detection results across IR, RGB, and fused RGB+IR modalities against Ground Truth.

Key Results and Performance

The effectiveness of SuperYOLO was primarily benchmarked on the VEDAI dataset, which contains 1,246 images with both RGB and IR modalities across 11 vehicle classes.

  • Accuracy: SuperYOLO achieved 75.09% mAP50 on VEDAI, outperforming larger state-of-the-art models like YOLOv5x and the specialized YOLOrs by more than 10%.
  • Efficiency: Despite its superior accuracy, SuperYOLO has 18x fewer parameters and 3.8x fewer GFLOPs than YOLOv5x.
  • Speed: By removing the SR branch after training, the model maintains the high inference speed of the baseline YOLOv5s, making it ideal for real-time on-board processing on satellites or drones.
  • Generalization: On single-modality datasets like DOTA, SuperYOLO achieved 69.99% mAP, demonstrating its robustness even when IR data is unavailable.

Conclusion

SuperYOLO demonstrates that for small object detection, maintaining spatial resolution and leveraging auxiliary training tasks can be more effective than simply increasing network depth. By combining efficient pixel-level multimodal fusion with training-time super-resolution supervision, it provides a state-of-the-art solution for real-time remote sensing applications. Future work may explore more advanced SR architectures and refined fusion techniques to further enhance performance in noisy environments.

Summary updated

BibTeX
@article{zhang2023superyolo,
  author = {Jiaqing Zhang and Jie Lei and Weiying Xie and Zhenman Fang and Yunsong Li and Qian Du},
  title = {{SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2023},
  doi = {10.1109/tgrs.2023.3258666},
}
Figure