Intelligent Object Recognition of Urban Water Bodies Based on Deep Learning for Multi-Source and Multi-Temporal High Spatial Resolution Remote Sensing Imagery

Shiran Song, Jianhua Liu, Yuan Liu, Guoqiang Feng, Hui Han, Yuan Yao, Mingyi DuSensors 2020 · DOI

TL;DR

This paper presents a novel deep learning approach, a modified Mask R-CNN, for accurately identifying urban water bodies in high-resolution remote sensing images. By integrating multi-source and multi-temporal data, their method demonstrates robust performance, effectively distinguishing water from shadows and other features, and offers a data-driven solution for improved water resource management.

What Problem Does This Paper Address?

Accurate and timely mapping of urban water bodies is crucial for understanding environmental changes, managing water resources, and disaster response. Traditional remote sensing methods often struggle with the complexity of high-resolution imagery, which includes diverse water body types, varying spectral characteristics due to water quality and surrounding environments, and confusion with shadows from buildings and vegetation. Existing methods often rely on manual feature selection or thresholding, which can be time-consuming, inefficient, and lack generalization across different sensors and times. While Geographic Object-Based Image Analysis (GEOBIA) has improved, it often overlooks top-down feedback mechanisms that can refine recognition. Deep learning offers powerful feature extraction capabilities, but adapting it effectively to the nuances of high-resolution, multi-source, and multi-temporal remote sensing data for water body recognition remains a challenge.

Methods

The core of this research is a modified Mask R-CNN architecture. Mask R-CNN is a state-of-the-art deep learning framework for object detection and instance segmentation. The authors adapted it by:

  • Data Fusion: Employing Principal Component Substitution (PCS) to fuse panchromatic and multispectral bands from WorldView-3 and GF-2 satellites, creating high spatial and spectral resolution images.
  • Modified Mask R-CNN: While the paper doesn't detail specific architectural modifications beyond stating it integrates "bottom-up and top-down processes," they emphasize enhancing the extraction of middle and high-level features. They also experimented with different ResNet backbone depths (50, 65, 80, 110, and a custom 116-layer ResNet) to find the optimal balance between feature extraction and spatial resolution preservation.
  • Training Strategy: The model was trained using a large dataset of manually labeled water body samples. They also explored cross-validation across different datasets to assess generalization.
  • Data Pre-processing: Images were sliced into 500x500 pixel tiles, linearly stretched (2%-98% histogram distribution), and bit-depth reduced to 8 bits to optimize processing speed and fit the Mask R-CNN input requirements.

Datasets

The study utilized high spatial resolution remote sensing imagery (HSRRSI) of Tongzhou New Town, Beijing. Specifically:

  • WorldView-3: Data from September 2017 and October 2014.
  • GF-2: Data from April 2016 and September 2018.

These datasets provided multi-source (different sensors) and multi-temporal (different acquisition dates) imagery. The authors manually extracted and labeled water samples to create training and testing datasets.

Key Insights and Ideas

  • Deep Learning for Water Bodies: The study confirms that deep learning, specifically Mask R-CNN, is highly effective for urban water body recognition, outperforming traditional machine learning methods.
  • Data-Driven Approach: The proposed method is entirely data-driven, eliminating the need for manual feature engineering or prior knowledge, which is a significant advantage for automation and generalization.
  • Handling Shadows: The Mask R-CNN approach effectively mitigates confusion with shadows from buildings and vegetation, a common challenge in remote sensing.
  • Multi-Source/Multi-Temporal Robustness: The model demonstrates strong generalization capabilities, performing well across different sensors and acquisition times, indicating its potential for broad application.
  • Network Depth Trade-off: While deeper networks generally improve accuracy, the study found that excessively deep networks (like ResNet 110) could lead to a loss of spatial resolution, negatively impacting recognition of smaller features. A custom 116-layer ResNet was found to offer the best performance.

Key Results

  • The proposed Mask R-CNN method achieved high accuracy in water body recognition across four different datasets.
  • Average precision of 0.85 and average recall of 0.97 were reported when training and testing data were from the same dataset.
  • Even with cross-validation (different training and testing areas), the method maintained a respectable 0.80 average precision and 0.90 average recall, demonstrating its generalization ability.
  • Compared to traditional methods (Cart, SVM, KNN, Random Trees), Mask R-CNN consistently outperformed them in precision and recall across all test areas. For example, in the WV-2014 dataset, Mask R-CNN achieved 0.9440 precision and 0.9752 recall, significantly higher than the next best method (Random Trees with 0.9360 precision and 0.9669 recall).

Limitations and Future Work

  • Limitations: The paper acknowledges that while the method shows strong generalization, achieving the standards of surveying and mapping products is still a long-term goal. The performance might still be affected by extremely complex scenes or very subtle water features.
  • Future Work: The authors suggest extending this method to rapid data analysis for flooding and other thematic information. They also propose combining these techniques with digital terrain models to produce volumetric estimates of inundated areas, which would be valuable for flood analysis. Further research will focus on making extraction more "active" and intelligent, and on continuous monitoring and updating of water bodies using accumulated observations.

Summary updated

Abstract

High spatial resolution remote sensing image (HSRRSI) data provide rich texture, geometric structure, and spatial distribution information for surface water bodies. The rich detail information provides better representation of the internal components of each object category and better reflects the relationships between adjacent objects. In this context, recognition methods such as geographic object-based image analysis (GEOBIA) have improved significantly. However, these methods focus mainly on bottom-up classifications from visual features to semantic categories, but ignore top-down feedback which can optimize recognition results. In recent years, deep learning has been applied in the field of remote sensing measurements because of its powerful feature extraction ability. A special convolutional neural network (CNN) based region proposal generation and object detection integrated framework has greatly improved the performance of object detection for HSRRSI, which provides a new method for water body recognition based on remote sensing data. This study uses the excellent "self-learning ability" of deep learning to construct a modified structure of the Mask R-CNN method which integrates bottom-up and top-down processes for water recognition. Compared with traditional methods, our method is completely data-driven without prior knowledge, and it can be regarded as a novel technical procedure for water body recognition in practical engineering application. Experimental results indicate that the method produces accurate recognition results for multi-source and multi-temporal water bodies, and can effectively avoid confusion with shadows and other ground features.
BibTeX
@inproceedings{song2020intelligent,
  author = {Shiran Song and Jianhua Liu and Yuan Liu and Guoqiang Feng and Hui Han and Yuan Yao and Mingyi Du},
  title = {{Intelligent Object Recognition of Urban Water Bodies Based on Deep Learning for Multi-Source and Multi-Temporal High Spatial Resolution Remote Sensing Imagery}},
  booktitle = {Sensors},
  year = {2020},
  doi = {10.3390/s20020397},
}
Figure