Context Aggregation Network for Semantic Labeling in Aerial Images

Wensheng Cheng, Wen Yang, Min Wang, Gang Wang, Jinyong ChenRemote 2019 · DOI

TL;DR

This paper introduces the Context Aggregation Network (CAN), a novel end-to-end deep learning model designed for semantic labeling of aerial images. CAN effectively addresses the challenges of multi-scale object recognition and precise localization in high-resolution aerial imagery by incorporating novel Context Fuse and Attention Mix modules, achieving state-of-the-art performance on benchmark datasets.

What Problem Does This Paper Address?

Semantic labeling, or assigning a category to each pixel in an image, is crucial for analyzing aerial imagery for applications like land-use surveys and environmental monitoring. However, aerial images present unique challenges: objects vary drastically in size (from tiny cars to large buildings), and scenes are complex with interwoven elements like trees and vegetation. Traditional methods struggle with these variations, and even existing Convolutional Neural Network (CNN) approaches face difficulties in accurately recognizing objects at different scales and precisely localizing their boundaries. This paper aims to overcome these limitations by developing a more robust and context-aware semantic labeling network.

Methods

The core of the proposed solution is the Context Aggregation Network (CAN), an encoder-decoder architecture built upon a ResNet-50 backbone. CAN integrates three novel modules:

  • Context Fuse Module (CFM): This module is designed to aggregate multi-scale contextual information. It comprises parallel convolutional branches with varying kernel sizes (e.g., 3x3, 7x7, 11x11, 15x15) to capture features at different receptive fields. Additionally, a global pooling branch is included to incorporate global context, which is vital for understanding the overall scene. Unlike some prior methods that use dilated convolutions (which can lead to gridding artifacts) or spatial pyramid pooling (which can lose detail), CFM uses standard convolutions and a global pooling strategy for more effective context aggregation.
  • Attention Mix Module (AMM): This module focuses on effectively fusing multi-level features. It leverages a channel-wise attention mechanism to combine low-level (high-resolution, structural) and high-level (low-resolution, semantic) features. By adaptively emphasizing more discriminative feature channels, AMM enhances localization accuracy and the ability to distinguish between similar-looking objects. This approach goes beyond simple concatenation or addition of features, considering inter-channel dependencies.
  • Residual Convolutional Module (RCM): This module is used to refine features at all levels. Inspired by residual learning principles, RCM helps to adapt features for the semantic labeling task and makes the network deeper, thereby improving its capacity to learn sophisticated features.

The CAN architecture uses ResNet-50 as the encoder, with RCM applied to features from each stage. The final feature map from ResNet is then fed into the CFM to capture multi-scale context. The outputs from different ResNet stages are combined using the AMM, and the resulting features are refined by another RCM before the final semantic labeling prediction.

Datasets

The proposed CAN model was evaluated on two publicly available benchmark datasets for semantic labeling of aerial images:

  • ISPRS Vaihingen Dataset: This dataset contains 3-band IRRG images with a Ground Surface Distance (GSD) of approximately 9 cm. It includes 33 images with 5 labeled categories: impervious surface, building, low vegetation, tree, and car. For evaluation, only the 3-band IRRG images were used, excluding Digital Surface Model (DSM) and Normalized Digital Surface Model (NDSM) data to demonstrate the model's capability using only image data.
  • ISPRS Potsdam Dataset: This dataset features 4-band IRRGB images with a GSD of about 5 cm. It comprises 38 images, also with 5 labeled categories identical to the Vaihingen dataset. Similar to Vaihingen, only the 3-band IRRG subset of the images was used for training and evaluation.

The paper describes both local and benchmark evaluations for these datasets, with specific data splits for training and testing provided.

Key Insights and Ideas

  • Context is King (Especially in Aerial Imagery): The paper strongly emphasizes that aggregating context at multiple scales is crucial for accurate semantic labeling in aerial images, given their inherent scale variations and complex scenes.
  • Beyond Simple Feature Fusion: The Attention Mix Module's use of channel-wise attention to fuse multi-level features is a key innovation. It suggests that simply combining features isn't enough; adaptively weighting channels based on their discriminative power leads to better results.
  • Standard Convolutions Can Be Powerful: The CFM's use of standard convolutional layers with varying kernel sizes, rather than dilated convolutions, effectively captures multi-scale context without introducing artifacts like gridding.
  • End-to-End Simplicity: The proposed CAN is an end-to-end network, simplifying the pipeline and avoiding the need for separate post-processing steps like Conditional Random Fields (CRFs) that are common in other methods.
  • Raw Image Data Sufficiency: A significant finding is that the proposed model achieves state-of-the-art performance using only raw image data, without relying on auxiliary data like DSM, which is often a prerequisite for other high-performing models. This broadens its applicability.

Key Results

The CAN model demonstrated superior performance on both the ISPRS Vaihingen and Potsdam datasets.

  • Local Evaluation: On the Vaihingen dataset, CAN achieved an overall accuracy of 91.2% and an average IoU of 82.6%, outperforming leading models like DeepLabv3+ (87.2% accuracy, 81.2% IoU) and CASIA (90.6% accuracy, 84.3% IoU). Similar improvements were observed on the Potsdam dataset, with CAN achieving 90.9% overall accuracy and 78.2% average IoU, surpassing DeepLabv3+ and other benchmark methods.
  • Benchmark Evaluation: Even when compared against methods that utilize DSM/NDSM data, CAN, using only raw image data, achieved competitive or superior results. For instance, on the Vaihingen dataset, CAN achieved an overall accuracy of 91.2%, outperforming CASIA (90.6%) which also used DSM data.
  • Ablation Study: The ablation study confirmed the effectiveness of each proposed module. Starting with a base ResNet-50 model (69.3% mean IoU), adding RCM improved it to 75.9%, CFM further boosted it to 76.5%, and the final addition of AMM resulted in 76.8% mean IoU.

Limitations and Future Work

  • Limitations:
    • The paper notes that for local evaluation, full label images were used to set a higher standard, while for benchmark evaluation, eroded label images were used to align with existing methods. This difference in ground truth processing could be a point of consideration.
  • The model's performance is evaluated using only 3-band IRRG images, and its performance with additional spectral bands or DSM data is not explored.
  • Future Work:
    • The authors suggest that their model's ability to combine multi-level features more efficiently and exploit features more extensively opens avenues for further improvements.
  • The paper's focus on raw image data suggests future work could explore how to best integrate auxiliary data like DSM or other spectral bands with CAN to potentially achieve even higher performance.
  • The authors have made their PyTorch implementation publicly available, which facilitates further research and development in this area.

Summary updated

Abstract

Semantic labeling for high resolution aerial images is a fundamental and necessary task in remote sensing image analysis. It is widely used in land-use surveys, change detection, and environmental protection. Recent researches reveal the superiority of Convolutional Neural Networks (CNNs) in this task. However, multi-scale object recognition and accurate object localization are two major problems for semantic labeling methods based on CNNs in high resolution aerial images. To handle these problems, we design a Context Fuse Module, which is composed of parallel convolutional layers with kernels of different sizes and a global pooling branch, to aggregate context information at multiple scales. We propose an Attention Mix Module, which utilizes a channel-wise attention mechanism to combine multi-level features for higher localization accuracy. We further employ a Residual Convolutional Module to refine features in all feature levels. Based on these modules, we construct a new end-to-end network for semantic labeling in aerial images. We evaluate the proposed network on the ISPRS Vaihingen and Potsdam datasets. Experimental results demonstrate that our network outperforms other competitors on both datasets with only raw image data.
BibTeX
@inproceedings{cheng2019context,
  author = {Wensheng Cheng and Wen Yang and Min Wang and Gang Wang and Jinyong Chen},
  title = {{Context Aggregation Network for Semantic Labeling in Aerial Images}},
  booktitle = {Remote},
  year = {2019},
  doi = {10.3390/rs11101158},
}
Figure