CHAI: Craters in Historical Aerial Images

Marvin Burges, Sebastian Zambanini, Philipp PirkerWACV 2024

TL;DR

This paper introduces CHAI, the first publicly available dataset of annotated bomb craters from World War II aerial imagery, addressing a critical gap in historical aerial image analysis. They demonstrate its utility by evaluating 15 object detection models, highlighting the challenges and establishing a benchmark for future research in automated crater detection and historical event analysis.

What Problem Does This Paper Address?

Understanding the impact of historical events, particularly wartime activities, often relies on analyzing historical aerial imagery. World War II bomb craters, for instance, provide crucial insights into bombing campaigns, potential unexploded ordnance (UXO) risks, and environmental damage. However, the scarcity of publicly accessible, annotated datasets for historical aerial images makes it difficult to develop and evaluate automated methods for detecting these features. Existing datasets are either for celestial bodies, lack annotations, or are not publicly available, hindering progress in this domain. This paper aims to bridge this gap by creating and releasing a valuable resource for researchers.

Methods

The core of this paper is the creation and release of the CHAI (Craters in Historical Aerial Images) dataset. The authors collected 99 historical aerial images from Austria and Germany dating back to World War II. These images were meticulously annotated by domain experts, identifying bomb craters. To facilitate research, the dataset was pre-processed into 960x960 pixel patches with a 20% overlap. The authors then evaluated the performance of 15 different object detection models, ranging from established architectures like Faster R-CNN to more recent ones like DINO, on this dataset. This evaluation serves to benchmark the dataset's difficulty and highlight the capabilities of current detection methods.

Datasets

  • CHAI Dataset: A newly introduced dataset comprising 99 historical aerial images from Austria and Germany (1943-1945) with 19,618 annotated bomb craters. The dataset is divided into training, validation, and test sets, and is provided in both full and "light" (crater-only patches) versions.
  • Existing Datasets (for context):
    • EuroSDR TIME dataset: Contains 941 unlabeled historical aerial images from Europe, with only 20 from WWII and none with crater annotations.
  • Celestial body datasets (Mars, Moon): While available, these are noted as being easier to detect craters in due to lack of vegetation and development, and training on them did not improve performance on Earth imagery.
  • General object detection datasets (COCO, PASCAL VOC, XView, FMOW, DOTA, DIOR): These are for common objects in modern imagery and not directly applicable to historical aerial images.

Key Insights and Ideas

  • The Value of Historical Data: The paper strongly emphasizes the underappreciated value of historical aerial imagery for understanding past events and their lasting impact, particularly concerning UXO risks.
  • Annotation is Key (and Hard): The creation of the CHAI dataset highlights the significant effort and expertise required for accurate annotation of historical aerial images, especially for features like bomb craters where context and domain knowledge are crucial.
  • CHAI as a Benchmark: The dataset is positioned as a crucial benchmark for developing and evaluating object detection algorithms specifically for historical aerial imagery, a niche that has been underserved.
  • DINO's Dominance: Among the tested models, DINO emerged as the top performer, demonstrating its strong capabilities even on this challenging, specialized dataset.
  • Colorization's Subtle Impact: Experimenting with colorizing grayscale images showed a minimal impact on overall detection performance but notably improved AP75 scores, suggesting potential benefits for specific metrics, albeit at the cost of inference speed.

Key Results

  • Dataset Creation: The CHAI dataset contains 99 images with 19,618 annotated bomb craters, split into training, validation, and test sets.
  • Model Performance: The DINO object detector achieved the highest overall performance across various metrics on the CHAI dataset. Other models like Condi. DETR and YoloX-1 also showed competitive results.
  • Training Efficiency: Training exclusively on the "light" dataset (patches with at least one crater) was found to be sufficient and significantly faster (59% reduction in training time) compared to using the full dataset.
  • Colorization Benefits: Colorizing grayscale images with Hyper-U-Net led to a ~30% improvement in AP75 scores for DINO, though it increased inference time.
  • Visual Analysis: The DINO model performed well on urban scenes but struggled with complex industrial sites and images with snow cover, indicating areas for improvement.

Limitations and Future Work

  • Domain Gaps: The authors acknowledge inherent limitations due to significant domain gaps, such as snow coverage or urban environments, which may compromise the generalizability of trained algorithms.
  • Annotation Variability: Inconsistent annotation styles among human annotators were encountered, requiring manual correction.
  • Generalizability: The dataset's focus on WWII craters in Central Europe might limit direct applicability to other historical periods or geographical regions without domain adaptation.
  • Future Work: The paper suggests extending and enriching the dataset, developing methods to interpret algorithm decisions for responsible AI, and encouraging the open sharing of code and contributions from the research community. The potential for domain adaptation to modern satellite imagery for crater detection is also highlighted.

Summary updated

BibTeX
@inproceedings{burges2024chai,
  author = {Marvin Burges and Sebastian Zambanini and Philipp Pirker},
  title = {{CHAI: Craters in Historical Aerial Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure