Frequency distribution-aware network based on discrete cosine transformation (DCT) for remote sensing image super resolution

Yunsong Li, Debao YuanPeerJ Comput. Sci 2024 · OpenReview

TL;DR

This paper introduces FDANet, a novel deep learning approach for enhancing the resolution of remote sensing images. By leveraging the Discrete Cosine Transform (DCT) to analyze frequency distribution patterns, FDANet effectively captures both local frequency similarities and global frequency features, leading to superior image reconstruction compared to existing methods.

What Problem Does This Paper Address?

Remote sensing images are crucial for various applications, but their resolution is often limited by hardware constraints. Image Super-Resolution (SR) techniques aim to generate high-resolution images from low-resolution ones. While current deep learning methods excel at capturing spatial "non-local" features (i.e., correlations between different regions in the spatial domain), they often overlook the valuable information contained in the frequency distribution of images. This oversight limits their performance, especially for complex remote sensing imagery where frequency characteristics can be highly informative. This paper addresses this gap by proposing a network that explicitly leverages frequency domain information.

Methods

The core of the proposed method, FDANet, lies in its two novel modules:

  • Frequency-Aware (FA) Module: This module focuses on extracting similarity characteristics of frequency distribution between different regions. It achieves this by:
  • Dividing the image into small patches (8x8 pixels).
  • Applying the Discrete Cosine Transform (DCT) to each patch. DCT naturally separates low-frequency (upper-left corner) from high-frequency (lower-right corner) components.
  • Rearranging the frequency feature matrix to group similar frequency components across different regions. This allows the network to learn correlations in frequency distribution.
  • Utilizing a "global feature fusion block" to combine these frequency features.
  • Finally, applying the Inverse DCT (IDCT) to reconstruct the image.
  • Global Frequency (GF) Feature Fusion Module: This module aims to extract non-local information of feature maps at different scales within the frequency domain. It uses a modified U-Net architecture where DCT and a global feature fusion block are integrated into each layer of the encoder. This allows the network to learn global frequency patterns across various scales with reduced computational cost compared to methods like Transformers.

The global feature fusion block itself employs a "row-column decoupling strategy" involving transposing and convolving feature maps to efficiently capture global spatial relationships within the frequency domain.

Datasets

The proposed FDANet was evaluated on two commonly used remote sensing datasets:

  • UC Merced Land-Use dataset: Contains 21 scene classes with 100 samples each, with a pixel resolution of 1 foot.
  • WHU-RS19 dataset: Collected from Google Earth, featuring 19 physical scene categories.

For experiments, 10 scenes per dataset were selected, with 400 training images and 50 testing images for each.

Key Insights and Ideas

  • Frequency Domain is Underexplored: The paper highlights that while spatial non-local features are well-studied, the frequency domain's potential for image super-resolution, particularly frequency distribution similarities between regions, has been largely ignored.
  • DCT as a Feature Extractor: The Discrete Cosine Transform is not just for compression; it's a powerful tool for revealing frequency energy distribution and facilitating feature extraction in the frequency domain.
  • Frequency Distribution Similarity Matters: The FA module's ability to group and analyze similar frequency components across different image regions is a key innovation, allowing the network to understand how frequencies vary and correlate spatially.
  • Global Frequency Features Enhance Reconstruction: The GF module's approach to capturing non-local frequency information at multiple scales within the frequency domain complements the FA module, leading to a more comprehensive understanding of image characteristics.
  • Efficiency in Global Feature Extraction: The proposed global feature fusion block is noted for its lower computational complexity compared to other global information extraction methods like Transformers.

Key Results

  • Superior Performance: FDANet consistently outperformed several state-of-the-art SR algorithms (IMDN, HSENet, LGCNet, FENet, MAN) on both the UC Merced Land-Use and WHU-RS19 datasets across various scale factors (2x, 3x, 4x) and noise levels (0, 3, 7).
  • Quantitative Gains: For example, on the UC Merced dataset with a 3x scale and 0 noise, FDANet achieved a PSNR of 27.83 dB and an SSIM of 0.784, outperforming the next best (HSENet) by a noticeable margin. Similar improvements were observed on the WHU-RS19 dataset.
  • Robustness: The algorithm demonstrated robustness to different scale factors and noise levels, indicating its reliability for real-world remote sensing applications.
  • Visual Quality: Qualitative results showed that FDANet produced sharper edges and better noise removal, effectively reconstructing image details that were lost in low-resolution inputs.

Limitations and Future Work

  • Image Blurring: The current FDANet does not explicitly consider more complex degradation issues like image blurring, which can significantly alter frequency distribution. Future work should investigate its applicability to such scenarios.
  • Unsupervised Learning: The authors suggest that future research could explore unsupervised methods to leverage unlabeled remote sensing data more effectively.
  • Complex Degradation Factors: Investigating the inclusion of other complex degradation factors, such as blurring kernels, in RSI SR tasks is a promising future direction.

Summary updated

Abstract

Single-image super-resolution technology based on deep learning is widely used in remote sensing. The non-local feature reflects the correlation information between different regions. Most neural networks extract various non-local information of images in the spatial domain but ignore the similarity characteristics of frequency distribution, which limits the performance of the algorithm. To solve this problem, we propose a frequency distribution aware network based on discrete cosine transformation for remote sensing image super-resolution. This network first proposes a frequency-aware module. This module can effectively extract the similarity characteristics of the frequency distribution between different regions by rearranging the frequency feature matrix of the image. A global frequency feature fusion module is also proposed. It can extract the non-local information of feature maps at different scales in the frequency domain with little computational cost. The experiments were on two commonly-used remote sensing datasets. The experimental results show that the proposed algorithm can effectively complete image reconstruction and performs better than some advanced super-resolution algorithms. The code is available at https://github.com/Liyszepc/FDANet.
BibTeX
@inproceedings{li2024frequency,
  author = {Yunsong Li and Debao Yuan},
  title = {{Frequency distribution-aware network based on discrete cosine transformation (DCT) for remote sensing image super resolution}},
  booktitle = {PeerJ Comput. Sci},
  year = {2024},
  url = {https://openreview.net/forum?id=YAahYLftjq},
}
Figure