CNN-Based Spectral Super-Resolution of Panchromatic Night-Time Light Imagery: City-Size-Associated Neighborhood Effects

Natalya A. Rybnikova, Evgeny M. Mirkes, Alexander N. Gorban

TL;DR

This paper introduces a Convolutional Neural Network (CNN) approach to enhance the spectral resolution of panchromatic night-time light (NTL) satellite imagery, enabling the estimation of RGB color information. The study highlights how the optimal input data size for this "spectral super-resolution" task is influenced by the geographical size of the metropolitan area being analyzed, suggesting a "neighborhood effect" that is crucial for accurate color prediction.

What Problem Does This Paper Address?

Satellite-based night-time light (NTL) data, primarily from sensors like VIIRS-DNB, provides valuable global information on human presence and activity. However, this data is typically panchromatic, offering only intensity information. Obtaining NTL data with spectral (color) information, specifically red, green, and blue (RGB) bands, is crucial for more nuanced analyses, as different NTL colors are linked to various economic activities, land-use types, and even potential health impacts. Unfortunately, high-resolution spectral NTL data is often locally available, sporadic, or commercially restricted, creating a significant gap for researchers. This paper aims to bridge this gap by developing a method to derive RGB NTL information from readily available panchromatic NTL data and auxiliary built-up area information.

Methods

The core of this research lies in applying Convolutional Neural Networks (CNNs) for spectral super-resolution. The authors frame the problem as learning a mapping from panchromatic NTL and built-up area data to RGB NTL data. Specifically, they:

  • Input Data Preparation: They create small, multi-layer "image patches" from panchromatic NTL and built-up area data. These patches serve as input to the CNN.
  • CNN Architecture: A "light-weight" CNN architecture is employed, consisting of 3D and 2D convolutional layers, followed by Batch Normalization, Rectified Linear Units, a fully connected layer, and a regression layer. This architecture is designed to be efficient given the relatively small input image sizes.
  • Neighborhood Effect Exploration: A key aspect of their methodology is investigating the "neighborhood effect." This involves experimenting with different input image patch sizes (from 3x3 to 15x15 pixels) to determine how much surrounding spatial information is beneficial for predicting the central pixel's RGB value.
  • Training and Validation: CNN models are trained for each metropolitan area individually and then validated on the remaining cities. This approach allows them to assess the generalizability of their models and the impact of city size.
  • Performance Metrics: Model performance is evaluated using Pearson's correlation coefficient (to assess relative accuracy) and Weighted Mean Squared Error (WMSE) (to assess absolute accuracy).

Datasets

The study utilizes the following datasets:

  • RGB NTL Images: Multispectral (RGB) images from the International Space Station (ISS) for eight metropolitan areas: Atlanta (US), Beijing (China), Haifa (Israel), Khabarovsk (Russia), London (UK), Naples (Italy), Nashville (US), and Tianjin (China). These are considered the ground truth for RGB NTL.
  • Panchromatic NTL Images: Spatially corresponding cropped areas from VIIRS-DNB monthly composite images, obtained from the Earth Observation Group site. These provide the primary NTL intensity data.
  • Built-up Area Data: Global raster layer of human built-up area and settlement extent (HBASE) database, reporting pixel-wise percentage of built-up area. This data is used as a proxy for land-use types and is sourced from the NASA Socioeconomic Data and Application Center.

Key Insights and Ideas

  • City Size Matters for Neighborhood Effects: The most significant finding is that the optimal input image size for spectral super-resolution is directly related to the geographical size of the metropolitan area. Smaller cities benefit from smaller input patches (e.g., 5x5), while larger cities require larger patches (e.g., 7x7 or more). This suggests that the spatial context of NTL is scale-dependent.
  • Color-Specific Neighborhoods: For larger cities, the optimal input image size can also vary depending on the color band being predicted. Blue light prediction, for instance, may benefit from smaller neighborhoods compared to red light prediction. This hints at different spatial distributions and characteristics of light sources for different colors.
  • CNNs Outperform Traditional ML in Stability: While CNNs achieved comparable Pearson's correlation to previously used machine learning techniques (linear regression, random forest, elastic map), they demonstrated significantly better and more stable performance in terms of WMSE, especially on testing datasets. This suggests CNNs are more robust to variations in the data.
  • Built-up Area as a Crucial Predictor: The inclusion of built-up area data, alongside panchromatic NTL, is vital. This data acts as a proxy for land-use types, which are known to influence NTL spectral properties.

Key Results

  • Comparable Correlation, Better WMSE: CNN models achieved comparable Pearson's correlation coefficients to previous machine learning methods. However, they showed significantly better performance in terms of WMSE, with ratios for CNNs being 2.20–2.95 compared to 3.76–22.78 for other techniques on testing datasets.
  • Optimal Input Size Findings:
    • For smaller metropolitan areas (e.g., Haifa, Khabarovsk, Nashville, Naples), a 5x5 input image size was optimal for predicting all RGB bands.
  • For larger metropolitan areas (e.g., Atlanta, Beijing, London, Tianjin), optimal input image sizes were generally larger than 7x7.
  • In larger cities, blue light prediction often favored smaller input sizes (5x5 or 7x7) compared to red light prediction (11x11 or 13x13).
  • Visual Similarity: Visual inspection of the restored RGB images showed a fair similarity to the original ISS RGB images, though the restored images appeared slightly more blurred.

Limitations and Future Work

  • Limited Number of Cities: The study was conducted on eight metropolitan areas. The authors suggest that future work could involve experiments with a larger number of cities to further validate the findings.
  • Assumption of Built-up Area as Proxy: The assumption that built-up area data accurately represents land-use types influencing NTL color is a simplification. While effective, more direct land-use classification data could potentially improve results.
  • Speculative Explanation for City Size Effects: The explanation for why city size influences optimal neighborhood size is speculative (larger physical extent of urban features). This warrants further explicit testing.
  • Inconsistent Tendencies: The observed tendency for larger input sizes for red lights and smaller for blue lights was not consistently observed for all large cities (e.g., London and Tianjin showed some deviations), suggesting the need for further investigation.
  • Alternative CNN Architectures: While a "light-weight" CNN was used, exploring more complex or different CNN architectures could potentially yield further improvements.
  • Data Heterogeneity: The authors note that CNN models performed worse in terms of Pearson's correlation on testing sets compared to training sets, suggesting that CNN architectures less sensitive to data heterogeneity might be beneficial.

Summary updated

Abstract

Data on artificial night-time light (NTL), emitted from the areas, and captured by satellites, are available at a global scale in panchromatic format. In the meantime, data on spectral properties of NTL give more information for further analysis. Such data, however, are available locally or on a commercial basis only. In our recent work, we examined several machine learning techniques, such as linear regression, kernel regression, random forest, and elastic map models, to convert the panchromatic NTL images into colored ones. We compared red, green, and blue light levels for eight geographical areas all over the world with panchromatic light intensities and characteristics of built-up extent from spatially corresponding pixels and their nearest neighbors. In the meantime, information from more distant neighboring pixels might improve the predictive power of models. In the present study, we explore this neighborhood effect using convolutional neural networks (CNN). The main outcome of our analysis is that the neighborhood effect goes in line with the geographical extent of metropolitan areas under analysis: For smaller areas, optimal input image size is smaller than for bigger ones. At that, for relatively large cities, the optimal input image size tends to differ for different colors, being on average higher for red and lower for blue lights. Compared to other machine learning techniques, CNN models emerged comparable in terms of Pearson’s correlation but showed performed better in terms of WMSE, especially for testing datasets.
BibTeX
@inproceedings{rybnikova2021cnn,
  author = {Natalya A. Rybnikova and Evgeny M. Mirkes and Alexander N. Gorban},
  title = {{CNN-Based Spectral Super-Resolution of Panchromatic Night-Time Light Imagery: City-Size-Associated Neighborhood Effects}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=oocu-NWx5y},
}

Loading related content...

Figure