Generating High Quality Visible Images from SAR Images Using CNNs

Puyang Wang, Vishal M. PatelarXiv 2018 · OpenReview

TL;DR

This paper introduces SAR-GAN, a novel deep learning approach that transforms noisy, grayscale Synthetic Aperture Radar (SAR) images into high-quality, color visible-like images. By combining a cascaded network for despeckling and colorization within a Generative Adversarial Network (GAN) framework, SAR-GAN effectively addresses the interpretability challenges of SAR imagery, outperforming existing methods.

What Problem Does This Paper Address?

Synthetic Aperture Radar (SAR) is a powerful imaging technology, capable of capturing detailed images day or night and through adverse weather. However, SAR images are notoriously difficult to interpret for two primary reasons:

  1. Speckle Noise: SAR images are heavily corrupted by multiplicative speckle noise, which arises from the constructive and destructive interference of radar signals reflecting off small objects. This noise significantly degrades image quality and obscures fine details.
  2. Grayscale Output: Processed SAR images are typically grayscale, lacking the color information that humans use to easily distinguish between different surface types (e.g., distinguishing between a grassy field and a sandy area).

These limitations hinder both human interpretation and the application of computer vision systems. While existing methods can reduce speckle, they often fail to provide the necessary color information for semantic understanding. This paper aims to bridge this gap by generating realistic, color visible-like images directly from noisy SAR inputs.

Methods

The core of the proposed solution is SAR-GAN, a deep learning architecture built upon a cascaded Generative Adversarial Network (GAN). It comprises two main sub-networks:

  • Despeckling Sub-network ($G_D$): This network is designed to remove the speckle noise from the input SAR image. It employs a novel "division residual" method. Instead of directly learning a mapping to a clean image, it learns to estimate the speckle component. The despeckled image is then obtained by dividing the noisy input by this estimated speckle. This approach allows for end-to-end training, unlike methods that require separate log-transformations. The architecture uses convolutional layers, batch normalization, and ReLU activations, with a final hyperbolic tangent layer.
  • Colorization Sub-network ($G_C$): This network takes the despeckled grayscale image and adds color information to generate a visible-like RGB image. It utilizes a symmetric encoder-decoder architecture with skip connections, inspired by successful image colorization techniques. Skip connections are crucial for preserving low-level information and ensuring that the output retains structural details from the input.

These two sub-networks form the generator $G = G_C \circ G_D$. The entire SAR-GAN is trained using a combined loss function that balances:

  • L1 Loss: For both despeckling ($L_D$) and colorization ($L_C$), this loss encourages pixel-wise similarity between the generated and ground truth images. It's effective for denoising but can lead to grayish outputs in colorization.
  • Adversarial Loss ($L_A$): This loss is introduced by a discriminator network ($D$) which tries to distinguish between real visible images and the generated ones. It pushes the generator to produce more realistic and visually appealing outputs, avoiding the "grayish" artifacts often seen with L1 loss alone.

The combined loss function allows for end-to-end training of the cascaded network.

Datasets

  • Training Data: The authors generated a dataset of 3292 image pairs. The visible images were collected from scraped Google Maps, and corresponding speckled SAR images were synthetically generated using the multiplicative noise model described in the paper. All images were resized to 512x512.
  • Evaluation Data: Experiments were conducted on both simulated SAR images (with varying noise levels L=1, 4, 10) and a real SAR image from RADARSAT-1.

Key Insights and Ideas

  • Cascaded GAN for SAR: The paper demonstrates the effectiveness of a cascaded GAN architecture for the specific problem of SAR-to-visible image translation. This modular approach allows for specialized networks to handle distinct challenges (despeckling and colorization).
  • Division Residual for Despeckling: The novel "division residual" layer in the despeckling sub-network is a key innovation. It enables end-to-end learning by directly modeling the multiplicative noise, avoiding complex pre-processing steps.
  • Balancing L1 and Adversarial Loss: The authors highlight the necessity of combining L1 loss (for fidelity) and adversarial loss (for realism) to achieve high-quality, interpretable SAR-to-visible translations. Relying solely on L1 loss leads to dull, grayish outputs, while adversarial loss alone can introduce artifacts.
  • Importance of Skip Connections: For the colorization sub-network, skip connections are shown to be vital for preserving structural details and ensuring that the generated visible image accurately reflects the underlying scene geometry present in the despeckled SAR image.

Key Results

  • Superior Despeckling: Quantitative results (Table I) show that SAR-GAN's despeckling sub-network outperforms state-of-the-art methods like SAR-BM3D, CNN, and pix2pix across various noise levels (L=1, 4, 10) in terms of PSNR, SSIM, UQI, and DG.
  • High-Quality Visible Images: Qualitative results (Figures 1, 5, and 6) demonstrate that SAR-GAN successfully generates high-quality, color visible-like images from both simulated and real SAR inputs. These generated images are visually richer and more interpretable than those produced by other methods.
  • Reduced Artifacts: Compared to methods like pix2pix, SAR-GAN produces outputs with fewer artifacts, attributed to its more effective despeckling component.
  • Faster Convergence: The cascaded structure is noted to lead to faster convergence during training.

Limitations and Future Work

  • Artifacts from L1 Loss: While SAR-GAN is superior, the paper acknowledges that methods relying solely on L1 loss (like CNN in Figure 5(e) and (f)) tend to produce outputs with gray color, indicating a limitation in capturing vibrant color distributions.
  • SAR-BM3D Detail Loss: The paper notes that while SAR-BM3D helps suppress speckle, it can come at the cost of losing some fine detail information.
  • Future Work: The paper implicitly suggests that further improvements in GAN architectures and loss functions could lead to even more realistic and artifact-free SAR-to-visible translations. The authors' focus on a cascaded approach also opens avenues for exploring more complex multi-stage processing pipelines.

Summary updated

Abstract

We propose a novel approach for generating high quality visible-like images from Synthetic Aperture Radar (SAR) images using Deep Convolutional Generative Adversarial Network (GAN) architectures. The proposed approach is based on a cascaded network of convolutional neural nets (CNNs) for despeckling and image colorization. The cascaded structure results in faster convergence during training and produces high quality visible images from the corresponding SAR images. Experimental results on both simulated and real SAR images show that the proposed method can produce visible-like images better compared to the recent state-of-the-art deep learning-based methods.
BibTeX
@misc{wang2018generating,
  author = {Puyang Wang and Vishal M. Patel},
  title = {{Generating High Quality Visible Images from SAR Images Using CNNs}},
  year = {2018},
  eprint = {1802.10036},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.10036},
}
Figure