NTIRE 2021 Multi-modal Aerial View Object Classification Challenge

Jerrick Liu, Nathan Inkawhich, Oliver Nina, Radu Timofte, Sahil Jain, Bob Lee, Yuru Duan, Wei Wei, Lei Zhang, Songzheng Xu, Yuxuan Sun, Jiaqi Tang, Xueli Geng, Mengru Ma, Gongzhe Li, Huanqia Cai, Chengxue Cai, Sol Cummings, Casian Miron, Alexandru Pasarica, Cheng-Yen Yang, Hung-Min Hsu, Jiarui Cai, Jie Mei, Chia-Ying Yeh, Jenq-Neng Hwang, Michael Xin, Zhongkai Shangguan, Zihe Zheng, Xu Yifei, Lehan Yang, Kele Xu, Min Feng

TL;DR

This paper introduces the first-ever Multi-modal Aerial View Object Classification (MAVOC) challenge, focusing on combining Electro-Optical (EO) and Synthetic Aperture Radar (SAR) imagery for improved target recognition. The challenge highlights the complementary strengths of these sensors and showcases significant accuracy gains (over 15%) compared to baselines, demonstrating the power of multi-modal approaches in aerial view object classification.

What Problem Does This Paper Address?

Automatic Target Recognition (ATR) in aerial imagery is a critical but challenging task. While significant progress has been made, aerial view ATR faces unique hurdles: lower target resolution, complex textures, and varying lighting conditions. Traditional ATR systems often rely on a single sensor modality, like EO (visible spectrum imagery) or SAR (radar imagery). EO provides human-interpretable images but is dependent on light, while SAR can operate in all weather and lighting conditions but offers less detail and is harder for humans to interpret. This paper addresses the need for more robust ATR systems by exploring how to effectively fuse information from both EO and SAR sensors to overcome the limitations of individual modalities and achieve superior classification performance.

Methods

The MAVOC challenge was designed with two tracks to encourage diverse approaches:

  • Track 1: SAR Classification: Participants were tasked with building a classifier that performs optimally on SAR imagery, with the option to use both EO and SAR data during training. This track aimed to improve SAR-specific classification, useful when only SAR data is available at test time.
  • Track 2: EO + SAR Classification: This track required participants to jointly leverage both EO and SAR image pairs for training and testing, aiming for the highest accuracy when both modalities are available.

The paper also details the methods employed by the top-performing teams. Common themes included:

  • Deep Learning Architectures: ResNet variants (e.g., ResNet152) and EfficientNet were frequently used, often with modifications.
  • Data Augmentation: Techniques like random cropping, flips, rotations, and CutMix were extensively used to increase data diversity and robustness.
  • Handling Imbalanced Data: Strategies such as under-sampling, class balancing in batches, and using class weights were crucial due to the naturally imbalanced nature of the dataset.
  • Feature Fusion: Combining features extracted from EO and SAR modalities was a key strategy for Track 2.
  • Semi-Supervised Learning: Techniques like "Noisy Student" were employed to leverage unlabeled data and improve generalization.

Datasets

The challenge introduced a new dataset specifically designed for multi-modal aerial view object classification. This dataset consists of paired EO and SAR image chips from 10 different object classes.

  • EO Chips: 31x31 pixels.
  • SAR Chips: Generally around 55x55 pixels, offering finer resolution than EO but with SAR-specific processing artifacts.

The dataset was divided into:

  • Training Set: Non-uniformly distributed classes, reflecting real-world scenarios.
  • Validation Set: Uniformly distributed classes with fewer samples per class.
  • Test Set: Uniformly distributed classes, mirroring the validation set's distribution for fair evaluation.

Key Insights and Ideas

  • Complementarity is Key: The significant accuracy improvements across both tracks strongly suggest that EO and SAR data are highly complementary, with each sensor providing unique information that enhances classification.
  • Long-Tailed Data is a Major Hurdle: The imbalanced nature of the dataset (e.g., "sedan" being over 79% of the data) presented a significant challenge, requiring sophisticated data handling and training strategies.
  • Beyond Simple Fusion: While feature fusion was a common approach, the top methods often incorporated more advanced techniques like data augmentation, semi-supervised learning, and careful handling of data imbalance to achieve state-of-the-art results.
  • ResNet and EfficientNet Dominate: These established deep learning architectures, when adapted and combined with appropriate training strategies, proved highly effective for this task.

Key Results

The challenge demonstrated substantial improvements over established baselines.

  • Overall Improvement: Both tracks saw accuracy improvements of more than 15% compared to the provided baselines.
  • Track 1 (SAR): The top-ranked team, MISL-SAR, achieved an accuracy of 34.62%, significantly outperforming the SAR baseline of 15.87%.
  • Track 2 (EO+SAR): The winning team, XD-IPIU, reached an accuracy of 46.85%, showcasing the power of combining both modalities.

Limitations and Future Work

  • Dataset Specificity: While the introduced dataset is valuable, its specific characteristics (e.g., chip size, sensor types) might limit direct transferability to other aerial imagery datasets.
  • Human Interpretability: SAR imagery remains less human-interpretable, posing challenges for direct analysis and potentially requiring more sophisticated feature extraction methods.
  • Computational Cost: Advanced techniques like extensive data augmentation and complex model architectures can be computationally intensive.

The paper implicitly suggests future work in developing more efficient and robust multi-modal fusion techniques, exploring novel architectures tailored for EO and SAR data, and potentially expanding the dataset to include more diverse scenarios and sensor types. The success of the challenge itself points towards continued research in this area.

Summary updated

Abstract

In this paper, we introduce the first Challenge on Multi-modal Aerial View Object Classification (MAVOC) in conjunction with the NTIRE 2021 workshop at CVPR. This challenge is composed of two different tracks using EO andSAR imagery. Both EO and SAR sensors possess different advantages and drawbacks. The purpose of this competition is to analyze how to use both sets of sensory information in complementary ways. We discuss the top methods submitted for this competition and evaluate their results on our blind test set. Our challenge results show significant improvement of more than 15% accuracy from our current baselines for each track of the competition
BibTeX
@inproceedings{liu2021ntire,
  author = {Jerrick Liu and Nathan Inkawhich and Oliver Nina and Radu Timofte and Sahil Jain and Bob Lee and Yuru Duan and Wei Wei and Lei Zhang and Songzheng Xu and Yuxuan Sun and Jiaqi Tang and Xueli Geng and Mengru Ma and Gongzhe Li and Huanqia Cai and Chengxue Cai and Sol Cummings and Casian Miron and Alexandru Pasarica and Cheng-Yen Yang and Hung-Min Hsu and Jiarui Cai and Jie Mei and Chia-Ying Yeh and Jenq-Neng Hwang and Michael Xin and Zhongkai Shangguan and Zihe Zheng and Xu Yifei and Lehan Yang and Kele Xu and Min Feng},
  title = {{NTIRE 2021 Multi-modal Aerial View Object Classification Challenge}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=RaY36Kx4BVl},
}
Figure