NTIRE 2021 Multi-Modal Aerial View Object Classification Challenge

Jerrick Liu, Nathan Inkawhich, Oliver Nina, Radu Timofte

Abstract

In this paper, we introduce the first Challenge on Multimodal Aerial View Object Classification (MAVOC) in conjunction with the NTIRE 2021 workshop at CVPR. This challenge is composed of two different tracks using EO and SAR imagery. Both EO and SAR sensors possess different advantages and drawbacks. The purpose of this competition is to analyze how to use both sets of sensory information in complementary ways. We discuss the top methods submitted for this competition and evaluate their results on our blind test set. Our challenge results show significant improvement of more than 15% accuracy from our current baselines for each track of the competition.

Introduction: The Challenge of Aerial Object Recognition

Automatic Target Recognition (ATR) from an aerial perspective is notoriously difficult. Targets such as cars or trucks often occupy only a tiny fraction of the image, resolutions are low, and environmental factors like lighting and weather can obscure critical details.

Traditionally, researchers rely on Electro-Optical (EO) sensors—standard cameras that capture visible light. While EO images are high-resolution and easy for humans to interpret, they are ineffective at night or in heavy cloud cover. Conversely, Synthetic Aperture Radar (SAR) can "see" through clouds and darkness, but the resulting images are noisy, lower resolution, and difficult for even experts to interpret.

Visual comparison of EO and SAR image chips across ten vehicle classes, demonstrating the difficulty of SAR interpretation.

The MAVOC challenge addresses the gap in multi-modal research, asking: How can we combine the strengths of EO and SAR to create more robust classification systems?

Methods and Winning Strategies

The challenge was split into two tracks to test different operational scenarios:

  1. Track 1 (SAR Classification): Participants trained models using both EO and SAR data but were tested only on SAR images. This simulates a scenario where high-quality optical data is available for training, but only radar is available during deployment (e.g., a night mission).
  2. Track 2 (EO + SAR Classification): Participants used pairs of EO and SAR images for both training and testing, focusing on joint feature fusion.

Most top teams moved away from simple baselines toward deeper architectures like ResNet-152 and EfficientNet. The winning Track 1 team (MISL-SAR) specifically designed their network to eliminate redundant information between modalities to improve feature learning. Other successful techniques included semi-supervised learning (the "Noisy Student" approach), multi-stage training to focus on minority classes, and modified data augmentation strategies like CutMix to preserve small targets.

The Unicorn Dataset

The challenge introduced the Unicorn Dataset, which consists of 285,772 training pairs. Each pair contains a 31x31 px EO chip and a corresponding ~55x55 px SAR chip. A significant hurdle for participants was the dataset's extreme class imbalance.

Distribution of vehicle classes in the Unicorn Dataset, highlighting the extreme class imbalance favoring sedans.

As shown in the distribution, the dataset is heavily skewed toward "sedans" (over 234,000 samples), while rarer classes like "buses" have only 624 samples. Teams had to employ sophisticated undersampling and oversampling techniques to prevent models from becoming biased toward the majority class.

Key Results and Insights

The challenge results demonstrated that EO data can act as a powerful "teacher" for SAR. Track 1 showed that even if EO data isn't available at test time, using it during training significantly boosts a SAR-only classifier's performance. The optical data helps the model learn more distinct features that it can then identify in the noisier radar data.

Performance leaps over the provided AlexNet baselines were substantial:

  • Track 1: Baseline accuracy was 15.87%, while the winning team (MISL-SAR) achieved 34.62%.
  • Track 2: Baseline accuracy was 19.23%, while the winning team (XD-IPIU) reached 46.85%.

Conclusion

While the NTIRE 2021 challenge achieved a state-of-the-art advancement of over 15% in accuracy, several challenges remain. The dominance of the "sedan" class continues to hinder generalization, and the authors suggest that more sophisticated, fundamentally new multi-modal architectures are needed to move beyond simple feature concatenation. Future work will also need to address the transition from classification on pre-cropped chips to detection in wide-area imagery.

Summary updated

Abstract

In this paper, we introduce the first Challenge on Multimodal Aerial View Object Classification (MAVOC) in conjunction with the NTIRE 2021 workshop at CVPR. This challenge is composed of two different tracks using EO and SAR imagery. Both EO and SAR sensors possess different advantages and drawbacks. The purpose of this competition is to analyze how to use both sets of sensory information in complementary ways. We discuss the top methods submitted for this competition and evaluate their results on our blind test set. Our challenge results show significant improvement of more than 15% accuracy from our current baselines for each track of the competition.
BibTeX
@inproceedings{liu2021ntire,
  author = {Jerrick Liu and Nathan Inkawhich and Oliver Nina and Radu Timofte},
  title = {{NTIRE 2021 Multi-Modal Aerial View Object Classification Challenge}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=RaY36Kx4BVl},
}
Figure