High-Resolution Remote Sensing Data Classification over Urban Areas Using Random Forest Ensemble and Fully Connected Conditional Random Field

Xiaofeng Sun, Xiangguo Lin, Shuhan Shen, Zhanyi HuISPRS Int. J. Geo Inf 2017 · DOI

TL;DR

This paper introduces a novel three-stage approach for classifying high-resolution urban remote sensing data, combining multispectral imagery and 3D geometric data. By employing a Random Forest ensemble for classification and a Fully Connected Conditional Random Field for refinement, the method achieves state-of-the-art results among non-CNN approaches, demonstrating the power of fusing diverse data sources and advanced machine learning techniques.

What Problem Does This Paper Address?

Classifying high-resolution remote sensing data, especially in complex urban environments, is a long-standing challenge. The goal is to accurately assign a land cover class to each pixel, creating detailed maps for applications like urban planning and infrastructure management. While deep learning (CNNs) has shown impressive results, traditional methods often struggle with large datasets and capturing contextual information. This paper aims to bridge this gap by developing a robust, non-CNN-based framework that leverages both spectral and geometric data, and effectively incorporates contextual information for improved accuracy.

Methods

The proposed method consists of three main stages:

  • Feature Extraction: This stage combines information from multispectral images and 3D geometry data.
  • Spectral Features: Includes original RGB color space, HSV and CIE Lab color spaces (chosen for their advantages in representing color and perceptual uniformity), and the Normalized Difference Vegetation Index (NDVI).
  • Texture Features: Local range, standard deviation, and entropy are computed from grayscale versions of the multispectral images.
  • Height-Related Features: Digital Surface Model (DSM) and Normalized DSM (NDSM) values are used, along with texture features derived from the DSM (height-based texture).
  • Differential Morphological Profile (DMP) Features: These features are extracted from the DSM using morphological operations to capture shape and size information. In total, 24 features are extracted per pixel.

  • Classification (Random Forest Ensemble - RFE): To handle large datasets and avoid overfitting, an ensemble of Random Forest (RF) classifiers is used.

  • The training data is partitioned into tiles.
  • Individual RF models are trained on these tiles.
  • A validation set is used to fuse these individual RF models using a Bayesian weighted average, where each model's contribution is weighted by its accuracy. This creates the RFE model.
  • Classified Result Refinement (Fully Connected Conditional Random Field - FCCRF): To incorporate contextual information and smooth the classification results, an improved FCCRF model is employed.
  • Unlike traditional short-range CRFs, FCCRF models connect every pair of pixels, allowing for long-range dependencies.
  • The unary potential is derived from the class probabilities generated by the RFE model.
  • The pairwise potential uses feature vectors constructed from the most important features identified by the RFE model.
  • A mean-field approximate inference algorithm is used to optimize the FCCRF energy function.

Datasets

The primary dataset used for training and evaluation is the ISPRS Semantic Labeling Contest dataset from Vaihingen, Germany. This dataset includes:

  • High-resolution true orthophoto (TOP) mosaics with near-infrared, red, and green bands.
  • Digital Surface Models (DSM) generated using dense image matching.
  • Manually labeled ground truth data for 33 tiles, covering six land cover categories: impervious surfaces, building, grass, tree, car, and clutter/background. The dataset was divided into training, validation, and testing sets, with 16 tiles available for training and evaluation.

Key Insights and Ideas

  • Complementarity of Data Sources: Both multispectral imagery and 3D geometric data are crucial for accurate urban classification. While multispectral features (especially NDSM and NDVI) showed higher importance, the 3D data provides indispensable information.
  • Ensemble Learning for Robustness: Using an ensemble of Random Forest classifiers (RFE) effectively mitigates overfitting issues seen in individual RF models and improves overall classification accuracy and generalization capability.
  • Context Matters: The FCCRF refinement stage significantly improves classification results by incorporating spatial context, smoothing noisy regions, and better delineating object boundaries, which is crucial for urban scenes.
  • Feature Importance: NDSM and NDVI were identified as the most important features. Interestingly, multispectral image-derived features contributed more (69%) than 3D geometry features (31%) to the overall classification.
  • FCCRF Superiority: The proposed FCCRF model, by considering long-range dependencies and utilizing feature importance, achieved better refinement results than classical short-range CRF methods.

Key Results

  • The proposed three-stage method achieved an overall accuracy of 86.9% on the ISPRS Semantic Labeling Contest dataset.
  • This accuracy represents the state-of-the-art among non-CNN-based methods submitted to the benchmark.
  • The method ranked first in "imp surf" and "tree" classes, and second in "building," "grass," and "car" classes.
  • The FCCRF refinement stage provided a maximum accuracy increment of 0.88% with optimized hyper-parameters, compared to 0.57% for classical short-range CRF.
  • The 'car' class remains challenging, with an accuracy of 54.3%, attributed to inter-category differences and class imbalance.

Limitations and Future Work

  • 'Car' Class Classification: The low accuracy for the 'car' class is acknowledged as a limitation, potentially due to its small size, subtle height differences, and class imbalance. The authors suggest CNN-based methods might be better suited for this specific class.
  • Computational Time: The RFE classification stage can be time-consuming, especially with a large number of RF classifiers. This is noted as a drawback of ensemble methods.
  • Comparison to CNNs: While the method achieves state-of-the-art for non-CNN approaches, a gap remains compared to the best CNN-based methods.
  • Future Work: The authors suggest integrating CNNs as feature extraction submodules into their RFE framework to further boost performance and potentially outperform current best CNN methods. They also propose extending the method to other applications like vegetation mapping, water body mapping, and change detection.

Summary updated

BibTeX
@article{sun2017high,
  author = {Xiaofeng Sun and Xiangguo Lin and Shuhan Shen and Zhanyi Hu},
  title = {{High-Resolution Remote Sensing Data Classification over Urban Areas Using Random Forest Ensemble and Fully Connected Conditional Random Field}},
  journal = {ISPRS Int. J. Geo Inf.},
  year = {2017},
  doi = {10.3390/ijgi6080245},
}
Figure