Super-Resolution for "Jilin-1" Satellite Video Imagery via a Convolutional Network

Aoran Xiao, Zhongyuan Wang, Lei Wang, Yexian Ren

TL;DR

This paper introduces a novel, five-layer end-to-end convolutional neural network for super-resolving "Jilin-1" satellite video imagery. By eliminating pre- and post-processing steps and employing a joint loss function, the method achieves superior visual quality and quantitative metrics compared to existing approaches, making it more practical for real-world satellite applications.

What Problem Does This Paper Address?

Traditional Earth observation satellites capture still images, limiting their ability to monitor dynamic events. Newer video satellites offer continuous "staring" capabilities, but their imagery often suffers from lower spatial resolution due to imaging and transmission constraints. Super-resolution (SR) techniques aim to enhance this resolution. However, existing deep learning-based SR methods, like SRCNN and ESPCN, often require cumbersome pre-processing (e.g., bicubic interpolation) or post-processing (e.g., pixel rearrangement), which can introduce artifacts, increase complexity, and reduce performance. This paper tackles the challenge of developing an efficient and effective SR method specifically for satellite video that avoids these drawbacks and maintains the integrity of ground object representation.

Methods

The core contribution is a five-layer, end-to-end convolutional neural network designed for simplicity and efficiency. Key methodological aspects include:

  • End-to-End Architecture: The network operates directly on low-resolution (LR) input without any pre-processing steps like interpolation. This reduces computational cost and avoids introducing artifacts.
  • Novel Output Layers: Instead of relying on post-processing for pixel rearrangement (as in ESPCN), the proposed network incorporates either a reshape layer or a deconvolution layer at the end. This aims to better preserve the spatial distribution of ground objects within the image. Three variants are proposed: Mµ (baseline, similar to ESPCN's post-processing), MR (with reshape), and MD (with deconvolution).
  • Joint Loss Function: A unique loss function is formulated by combining the output loss (comparing the reconstructed image with the ground truth) and a loss derived from high-dimensional features of a non-linear mapping network. This joint approach allows the network to learn a more precise mapping relationship by considering features from earlier layers.
  • Satellite Video as Training Data: Crucially, the authors use "Jilin-1" satellite video data itself for training. This ensures consistency between training and testing data in terms of image content and statistical distribution, enhancing the method's practicality for its intended application.

Datasets

  • Training and Testing: The "Jilin-1" satellite video imagery is used for both training and testing.
  • Benchmark Comparison: For comparison, the authors also reference the "Yang91" dataset, a standard natural image set previously used for SR research.

Key Insights and Ideas

  • Eliminating Pre/Post-processing is Key: The paper strongly advocates for end-to-end networks that avoid pre- and post-processing. This not only simplifies the pipeline but also directly addresses performance bottlenecks and artifact introduction common in other methods.
  • Deconvolution for Spatial Integrity: The use of a deconvolution layer at the output shows promise for better preserving the spatial relationships of ground objects, which is critical for accurate remote sensing.
  • Joint Loss for Deeper Feature Learning: Combining output and feature-level losses allows the network to learn a more robust mapping, capturing both pixel-level accuracy and higher-level perceptual characteristics.
  • Self-Training for Practicality: Using the target domain's data (satellite video) for training is a significant practical advantage, circumventing the need for large, diverse, and potentially unavailable external datasets for remote sensing SR.

Key Results

  • Superior Performance: The proposed methods (MR and MD) consistently outperform baseline methods like bicubic interpolation, SCSR, and SRCNN (trained on both "Jilin-1" and "Yang91" datasets) in terms of both PSNR and SSIM metrics.
  • MD (Deconvolution) Leads: The MD variant, utilizing a deconvolution layer, generally achieves the best quantitative results and visual quality.
  • Visual Improvements: Qualitative results show that the proposed methods produce sharper images with better detail preservation and fewer artifacts compared to the baselines. SRCNN trained on the "Yang91" dataset shows limitations when applied to satellite video, highlighting the importance of domain-specific training.

Limitations and Future Work

  • Limitations:
    • The paper focuses on a specific five-layer architecture. While effective, exploring deeper or more complex network designs could potentially yield further improvements.
  • The paper mentions that the original SRCNN code had difficulty obtaining satisfactory results with video data, suggesting potential issues with its implementation or suitability for this specific data type.
  • Future Work:
    • The authors suggest that exploring deeper convolutional networks and more advanced loss functions could further enhance performance.
  • While the current method is end-to-end, further research into optimizing the network for real-time processing on satellite platforms could be valuable.

Summary updated

Abstract

Super-resolution for satellite video attaches much significance to earth observation accuracy, and the special imaging and transmission conditions on the video satellite pose great challenges to this task. The existing deep convolutional neural-network-based methods require pre-processing or post-processing to be adapted to a high-resolution size or pixel format, leading to reduced performance and extra complexity. To this end, this paper proposes a five-layer end-to-end network structure without any pre-processing and post-processing, but imposes a reshape or deconvolution layer at the end of the network to retain the distribution of ground objects within the image. Meanwhile, we formulate a joint loss function by combining the output and high-dimensional features of a non-linear mapping network to precisely learn the desirable mapping relationship between low-resolution images and their high-resolution counterparts. Also, we use satellite video data itself as a training set, which favors consistency between training and testing images and promotes the method’s practicality. Experimental results on “Jilin-1” satellite video imagery show that this method demonstrates a superior performance in terms of both visual effects and measure metrics over competing methods.
BibTeX
@inproceedings{xiao2018super,
  author = {Aoran Xiao and Zhongyuan Wang and Lei Wang and Yexian Ren},
  title = {{Super-Resolution for "Jilin-1" Satellite Video Imagery via a Convolutional Network}},
  booktitle = {Sensors},
  year = {2018},
  url = {https://openreview.net/forum?id=MQh884ShgGp},
}
Figure