Thermal to Visible Image Synthesis under Atmospheric Turbulence

Kangfu Mei, Yiqun Mei, Vishal M. Patel

Abstract

In many practical applications of long-range imaging such as biometrics and surveillance, thermal imagining modalities are often used to capture images in low-light and nighttime conditions. However, such imaging systems often suffer from atmospheric turbulence, which introduces severe blur and deformation artifacts to the captured images. Such an issue is unavoidable in long-range imaging and significantly decreases the face verification accuracy. In this paper, we first investigate the problem with a turbulence simulation method on real-world thermal images. An end-to-end reconstruction method is then proposed which can directly transform thermal images into visible-spectrum images by utilizing natural image priors based on a pre-trained StyleGAN2 network. Compared with the existing two-steps methods of consecutive turbulence mitigation and thermal to visible image translation, our method is demonstrated to be effective in terms of both the visual quality of the reconstructed results and face verification accuracy. Moreover, to the best of our knowledge, this is the first work that studies the problem of thermal to visible image translation under atmospheric turbulence.

Introduction

Long-range surveillance and biometric systems are critical for security in diverse environments. To maintain operational capability during nighttime or in low-light conditions, these systems frequently rely on thermal imaging. However, long-distance imaging introduces a significant physical challenge: atmospheric turbulence. This phenomenon, caused by variations in air temperature and pressure, creates a "shimmering" effect that severely degrades image quality through geometric distortion and blur.

When these degradations are applied to thermal images, the resulting data is often unusable for standard face recognition systems. This paper presents the first end-to-end framework designed to simultaneously mitigate atmospheric turbulence and translate thermal images into the visible spectrum, bridging the gap between degraded infrared captures and identifiable visible-light images.

The Challenge of Atmospheric Turbulence

Atmospheric turbulence impacts long-range imaging in two primary ways:

  1. Geometric Distortion: Random pixel shifts warp the structure of the face, making alignment and feature extraction difficult.
  2. Blur: High-frequency details—essential for identifying eyes, nose, and mouth shapes—are lost, resulting in a "smudged" appearance.

Traditional approaches to this problem typically employ a two-stage pipeline: first, a restoration step to mitigate turbulence in the thermal domain, followed by a translation step to convert the image to the visible spectrum. However, this sequential approach often suffers from "error propagation," where artifacts or failures in the restoration stage are amplified during the translation stage, leading to poor final results.

Proposed Methodology

We propose a novel GAN inversion network that treats restoration and translation as a single, unified task. By bypassing the two-stage pipeline, the model avoids the compounding errors of sequential processing.

Thermal-Turbulence Projection Module

The architecture begins with a multi-scale encoder-decoder network. Rather than compressing the input into a single latent code, this module extracts features at multiple resolutions. This allows the network to preserve global identity while capturing the local structural details necessary to counteract heavy distortion.

StyleGAN2-Based Reconstruction

To ensure the synthesized images are realistic and identifiable, we leverage a pre-trained StyleGAN2 network as a generative prior. StyleGAN2 possesses an inherent understanding of human facial structures. Our framework "modulates" the internal features of this pre-trained model using the data from the Projection Module. By modulating the GAN at every resolution level (from $4 \times 4$ to $1024 \times 1024$), the system can effectively correct both large-scale geometric warps and fine-scale blurs.

Training Objectives

The model is optimized using a combination of three critical loss functions:

  • Adversarial Loss: Ensures the output adheres to the distribution of real visible-light faces.
  • Perceptual Loss (VGG19): Maintains structural and semantic consistency between the input and output.
  • Identity Preserving Loss (LightCNN): Specifically ensures that the synthesized visible image maintains the unique identity of the subject captured in the thermal domain.

Experimental Evaluation

The framework was evaluated using two primary datasets modified with simulated atmospheric turbulence (simulating a 300-meter range):

  1. VIS-TH: Paired Long Wave Infrared (LWIR) and visible images of 50 subjects.
  2. ARL-VTF: A large-scale face verification dataset containing 220 identities.

Key Results

The proposed end-to-end method significantly outperformed both one-stage (e.g., HifaceGAN) and two-stage (e.g., TDRN + HifaceGAN) baselines.

  • Face Verification Accuracy: On the VIS-TH dataset, our method achieved a Rank-1 accuracy of 48.10%. This represents a massive improvement over the LightCNN baseline (18.10%) and the traditional two-stage approach (32.38%).
  • Image Quality: On the ARL-VTF dataset, the method achieved superior scores across key metrics, including LPIPS (0.2185) and PSNR (19.06), indicating higher visual fidelity and better structural similarity to ground-truth visible images.
  • Identity Preservation: Cosine distance measurements in the feature space confirmed that our method is significantly more effective at keeping subjects recognizable compared to direct translation methods.

Conclusion

This research demonstrates that performing restoration and modality translation simultaneously is superior to sequential processing for long-range thermal imaging. By utilizing the rich generative priors of StyleGAN2, the proposed framework can reconstruct clear, identifiable visible-spectrum faces from highly degraded thermal data. While future work will focus on testing against real-world (non-simulated) turbulence and reducing computational overhead, this study establishes a new benchmark for thermal-to-visible synthesis in challenging atmospheric conditions.

Summary updated

Abstract

In many practical applications of long-range imaging such as biometrics and surveillance, thermal imagining modalities are often used to capture images in low-light and nighttime conditions. However, such imaging systems often suffer from atmospheric turbulence, which introduces severe blur and deformation artifacts to the captured images. Such an issue is unavoidable in long-range imaging and significantly decreases the face verification accuracy. In this paper, we first investigate the problem with a turbulence simulation method on real-world thermal images. An end-to-end reconstruction method is then proposed which can directly transform thermal images into visible-spectrum images by utilizing natural image priors based on a pre-trained StyleGAN2 network. Compared with the existing two-steps methods of consecutive turbulence mitigation and thermal to visible image translation, our method is demonstrated to be effective in terms of both the visual quality of the reconstructed results and face verification accuracy. Moreover, to the best of our knowledge, this is the first work that studies the problem of thermal to visible image translation under atmospheric turbulence.
BibTeX
@inproceedings{mei2022thermal,
  author = {Kangfu Mei and Yiqun Mei and Vishal M. Patel},
  title = {{Thermal to Visible Image Synthesis under Atmospheric Turbulence}},
  booktitle = {ICIP},
  year = {2022},
  doi = {10.1109/icip46576.2022.9897975},
}
Figure