Object Detection in Aerial Images: A Large-Scale Benchmark and Challenges

Jian Ding, Nan Xue, Gui-Song Xia, Xiang Bai, Wen Yang, Michael Ying Yang, Serge J. Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, Liangpei Zhang

TL;DR

This paper introduces DOTA-v2.0, a massive new benchmark dataset for object detection in aerial images, featuring over 1.7 million oriented bounding box annotations. They also provide a comprehensive code library and extensive baseline evaluations to facilitate research in this challenging domain, which suffers from significant variations in scale, orientation, and object density compared to natural images.

What Problem Does This Paper Address?

Object detection in aerial images (ODAI) is crucial for applications like urban management and disaster relief, but it's significantly harder than in natural images. The primary challenges stem from the bird's-eye view, which leads to massive variations in object scale and orientation. Furthermore, the lack of large-scale, well-annotated benchmarks has been a major bottleneck, forcing researchers to adapt methods trained on natural image datasets, which often perform poorly due to domain differences. Existing aerial image datasets are often too small, lack precise oriented bounding box (OBB) annotations, or don't reflect the real-world complexities of object distribution and scale.

Methods

The authors introduce DOTA-v2.0, an expanded version of their previous DOTA dataset. Key methodological contributions include:

  • Large-Scale OBB Annotation: DOTA-v2.0 contains over 1.79 million object instances across 18 categories, all annotated with oriented bounding boxes (OBBs). This is crucial for accurately localizing objects with arbitrary orientations.
  • Comprehensive Baselines: They provide a unified code library (based on MMDetection) that supports ODAI, enabling fair comparisons. They evaluate over 70 configurations of 10 state-of-the-art algorithms, analyzing their speed and accuracy trade-offs.
  • OBB Representation: The paper emphasizes the importance of OBBs over horizontal bounding boxes (HBBs) for aerial images, detailing how to adapt existing detectors to predict OBBs. This involves modifying detection heads to regress OBB parameters.
  • Data Augmentation and Module Analysis: They conduct ablation studies on various data augmentation strategies (multi-scale training/testing, rotation) and analyze the impact of specific modules like RoI Transformer and deformable RoI pooling to understand their effectiveness in aerial image scenarios.

Datasets

  • DOTA-v2.0: The primary contribution, this dataset comprises 11,268 aerial images with 1,793,658 object instances across 18 categories, annotated with oriented bounding boxes (OBBs). It's sourced from Google Earth, GF-2 and JL-1 satellites, and airborne imagery.
  • Comparison Datasets: The paper compares DOTA-v2.0 with established natural image datasets like PASCAL VOC and MS COCO, as well as other aerial image datasets (e.g., VEDAI, DLR 3k, UCAS-AOD, xView, DIOR) to highlight DOTA's scale and annotation advantages.

Key Insights and Ideas

  • OBBs are Essential for Aerial Images: The arbitrary orientations in aerial imagery make OBBs significantly more effective than HBBs for accurate localization, especially for dense objects.
  • Domain Shift is Significant: Models trained on natural images struggle with aerial imagery due to differences in scale, orientation, and object density. This necessitates specialized benchmarks and algorithms.
  • Dataset Scale Matters: The sheer scale of DOTA-v2.0 (over 1.7 million instances) is critical for training robust deep learning models that can generalize across diverse aerial scenes and object types.
  • Hyperparameter Tuning is Crucial: Optimal settings for parameters like the number of proposals differ significantly between natural and aerial image datasets, with aerial images often requiring more proposals.
  • Geometric Transformations are Key: Modules like RoI Transformer and deformable pooling show promise in handling the geometric variations inherent in aerial images, though they don't fully solve the problem.

Key Results

  • DOTA-v2.0 Scale: The dataset is the largest public Earth vision object detection dataset to date, with over 1.79 million instances.
  • Baseline Performance: The paper provides extensive performance metrics (mAP) for 10 algorithms across DOTA-v1.0, v1.5, and v2.0. For instance, on DOTA-v2.0, the Faster R-CNN OBB + RoI Transformer achieved an OBB mAP of 53.37% with a speed of 12.4 fps.
  • Data Augmentation Impact: Both scale and rotation data augmentation significantly improve object detection performance on DOTA, highlighting their importance for handling variations.
  • Mask vs. OBB Head: Mask-based methods generally achieve better results but are more computationally expensive than OBB regression heads.

Limitations and Future Work

  • Inaccurate Annotations: The authors acknowledge potential errors in bounding box annotations (missing instances or inaccuracies) and suggest that future work could focus on modeling these inaccuracies, especially for OBBs.
  • Small Instance Instability: Extremely small instances can cause numerical instability during training, requiring filtering. While filtering had minimal impact on overall results, further research into handling these instances is suggested.
  • Generalization to Multi-Source: While DOTA-v2.0 includes diverse image sources, achieving strong generalization across all types of aerial imagery remains an ongoing challenge.
  • Further Dataset Extension: The authors plan to continue extending the dataset and hosting more challenges to drive research in ODAI.

Summary updated

Abstract

In the past decade, object detection has achieved significant progress in natural images but not in aerial images, due to the massive variations in the scale and orientation of objects caused by the bird's-eye view of aerial images. More importantly, the lack of large-scale benchmarks has become a major obstacle to the development of object detection in aerial images (ODAI). In this paper,we present a large-scale Dataset of Object deTection in Aerial images (DOTA) and comprehensive baselines for ODAI. The proposed DOTA dataset contains 1,793,658 object instances of 18 categories of oriented-bounding-box annotations collected from 11,268 aerial images. Based on this large-scale and well-annotated dataset, we build baselines covering 10 state-of-the-art algorithms with over 70 configurations, where the speed and accuracy performances of each model have been evaluated. Furthermore, we provide a code library for ODAI and build a website for evaluating different algorithms. Previous challenges run on DOTA have attracted more than 1300 teams worldwide. We believe that the expanded large-scale DOTA dataset, the extensive baselines, the code library and the challenges can facilitate the designs of robust algorithms and reproducible research on the problem of object detection in aerial images.
BibTeX
@article{ding2021object,
  author = {Jian Ding and Nan Xue and Gui-Song Xia and Xiang Bai and Wen Yang and Michael Ying Yang and Serge J. Belongie and Jiebo Luo and Mihai Datcu and Marcello Pelillo and Liangpei Zhang},
  title = {{Object Detection in Aerial Images: A Large-Scale Benchmark and Challenges}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2021},
  doi = {10.1109/tpami.2021.3117983},
}
Figure