Aerial Scene Understanding in The Wild: Multi-Scene Recognition via Prototype-based Memory Networks

Yuansheng Hua, Lichao Mou, Jianzhe Lin, Konrad Heidler, Xiao Xiang Zhu

TL;DR

PM-Net (Prototype-based Memory Network) tackles the challenge of recognizing multiple scenes within a single, unconstrained aerial image—a task far more complex than standard single-label classification. By leveraging an external memory to store "prototypes" learned from abundant single-scene data, the model can accurately interpret complex multi-scene environments even when specialized training data is scarce.

What Problem Does This Paper Address?

Most research in aerial scene recognition operates under a simplifying assumption: one image equals one label (e.g., "forest" or "airport"). However, real-world aerial imagery is rarely that tidy. A single high-resolution shot "in the wild" often contains a mixture of scenes, such as a residential area adjacent to a commercial zone and a parking lot.

Transitioning to multi-scene recognition presents two major hurdles:

  1. Annotation Complexity: Manually labeling multi-scene images is roughly ten times more time-consuming than single-scene labeling.
  2. Data Quality: Low-cost automated annotation sources, like OpenStreetMap (OSM), frequently suffer from incompleteness (missing labels) or incorrectness (mislabeling an orchard as residential).
Figure 2: Examples of incomplete (red) and incorrect (yellow) OSM data. Red: the com- mercial is not annotated in OSM data. Yellow: the orchard is mislabeled as residential.

The authors aim to bridge this gap by creating a model that can learn to understand complex, multi-label images using only a small amount of multi-scene training data, supplemented by massive amounts of existing, well-annotated single-scene data.

Methods

The authors propose PM-Net, which draws inspiration from how humans learn. Just as a child learns to recognize individual objects (like a "car" or a "tree") before understanding a complex scene (a "park with cars"), PM-Net learns individual scene prototypes first.

Figure 1: Illustration of how humans learn to perceive unconstrained aerial images being composed of multiple scenes. We first learn and memorize individual aerial scenes. Then we can possess the capability of understanding complex scenarios by learning from only a limited number of hard instances. We believe by simulating this learning process, a deep neural network can also learn to interpret multi-scene aerial images.

The architecture consists of three core components:

  1. Prototype Learning Module: A deep CNN (like ResNet or NASNet) acts as an embedding function $f_\phi$. It is trained on single-scene datasets to learn discriminative feature vectors. The average feature vector for each category becomes its "prototype."
  2. External Memory: These prototypes are stored in a matrix $M$. This acts as a knowledge base of what "pure" versions of each scene look like.
  3. Multi-head Attention-based Memory Retrieval: When a new multi-scene image is queried, the model uses a multi-head attention mechanism to compare the query's features against the stored prototypes. It retrieves the most relevant prototypes to make a final multi-label prediction.
Figure 3: The architecture of PM-Net, showing the flow from prototype learning to memory retrieval for multi-label prediction.

Datasets

The study utilizes three primary datasets:

  • MAI (Multi-scene Aerial Image) Dataset: A new benchmark introduced by the authors. it contains 3,923 large-scale images (512x512 pixels) with 24 scene categories, manually labeled to ensure high quality.
  • UCM Dataset: A standard single-scene dataset with 2,100 images across 21 classes.
  • AID Dataset: A large-scale single-scene dataset with 10,000 images across 30 classes.

The researchers created two experimental configurations, UCM2MAI and AID2MAI, to test how well prototypes learned from UCM or AID could generalize to the multi-scene MAI dataset.

Key Insights and Ideas

  • Leveraging Data Asymmetry: The "aha moment" is realizing that while multi-scene data is hard to get, single-scene data is plentiful. By treating single-scene features as "prototypes" in a memory bank, the model doesn't need to see every possible combination of scenes during training.
  • Multi-Head Attention for Scene Decomposition: Using multiple attention heads allows the model to focus on different parts of the image simultaneously. One head might find the "residential" prototype relevant to the top-left corner, while another identifies the "parking lot" prototype in the center.
  • Fine-tuning Matters: The authors found that while prototypes should be learned from single-scene data, the embedding function itself should remain trainable during the multi-scene phase to narrow the gap between "pure" and "mixed" image distributions.

Key Results

PM-Net consistently outperformed standard CNN baselines and existing memory network architectures (like Mem-N2N).

  • Performance Gains: On the UCM2MAI task, PM-NASNet achieved a mean $F_1$ score of 60.13%, a significant improvement over standard NASNet (52.03%) and Mem-N2N-NASNet (55.17%).
  • Robustness: The model proved capable of identifying small or peripheral scenes (like a bridge or a corner parking lot) that standard models often miss.
  • Efficiency: The system achieved these results using only a very limited number of multi-scene training instances (as few as 90-120 images), proving the effectiveness of the prototype-memory approach.

Limitations and Future Work

The authors noted that unseen scene appearances remain a challenge. For example, if the model learns "runway" and "apron" prototypes from clear-weather images, its performance drops significantly when presented with a multi-scene image where those areas are covered in snow.

Future work will focus on applying this prototype-based memory approach to weakly supervised scene recovery, potentially reducing the need for manual annotations even further.

Summary updated

Abstract

Aerial scene recognition is a fundamental visual task and has attracted an increasing research interest in the last few years. Most of current researches mainly deploy efforts to categorize an aerial image into one scene-level label, while in real-world scenarios, there often exist multiple scenes in a single image. Therefore, in this paper, we propose to take a step forward to a more practical and challenging task, namely multi-scene recognition in single images. Moreover, we note that manually yielding annotations for such a task is extraordinarily time- and labor-consuming. To address this, we propose a prototype-based memory network to recognize multiple scenes in a single image by leveraging massive well-annotated single-scene images. The proposed network consists of three key components: 1) a prototype learning module, 2) a prototype-inhabiting external memory, and 3) a multi-head attention-based memory retrieval module. To be more specific, we first learn the prototype representation of each aerial scene from single-scene aerial image datasets and store it in an external memory. Afterwards, a multi-head attention-based memory retrieval module is devised to retrieve scene prototypes relevant to query multi-scene images for final predictions. Notably, only a limited number of annotated multi-scene images are needed in the training phase. To facilitate the progress of aerial scene recognition, we produce a new multi-scene aerial image (MAI) dataset. Experimental results on variant dataset configurations demonstrate the effectiveness of our network. Our dataset and codes are publicly available.
BibTeX
@misc{hua2021aerial,
  author = {Yuansheng Hua and Lichao Mou and Jianzhe Lin and Konrad Heidler and Xiao Xiang Zhu},
  title = {{Aerial Scene Understanding in The Wild: Multi-Scene Recognition via Prototype-based Memory Networks}},
  year = {2021},
  eprint = {2104.11200},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.11200},
}
Figure