Thermal to Visible Image Synthesis Under Atmospheric Turbulence

Kangfu Mei, Yiqun Mei, Vishal M. Patel

TL;DR

This paper introduces the first end-to-end framework for synthesizing clear visible-spectrum face images from thermal images degraded by atmospheric turbulence. By leveraging the rich generative priors of a pre-trained StyleGAN2, the proposed method simultaneously mitigates turbulence artifacts (blur and distortion) and performs modality translation, significantly outperforming traditional two-step approaches in both image quality and face verification accuracy.

What Problem Does This Paper Address?

Long-range surveillance and biometric systems often rely on thermal imaging to operate in low-light or nighttime conditions. However, long-distance imaging is plagued by atmospheric turbulence—the shimmering effect caused by variations in air temperature and pressure. This turbulence introduces two major problems:

  1. Geometric Distortion: Pixels are shifted, warping the face.
  2. Blur: High-frequency details (like eyes, nose, and mouth shapes) are lost.

When these degradations are combined with the "domain gap" between thermal and visible light, standard face recognition systems fail. Previous attempts usually tried a two-step process: first "fixing" the turbulence in the thermal domain, then translating the result to visible light. This often leads to "error propagation," where artifacts from the first step ruin the second.

Methods

The authors propose a novel GAN inversion network that treats the problem as a single, unified task. The architecture consists of two main components:

  • Thermal-Turbulence Projection Module: A multi-scale encoder-decoder network. Instead of just producing a single "code," it extracts features at multiple resolutions from the degraded thermal image. This allows the network to capture both global identity and local structural details despite the heavy distortion.
  • Image Reconstruction Module (StyleGAN2): The researchers utilize a pre-trained StyleGAN2 network as a "prior." Since StyleGAN2 is already an expert at generating realistic faces, the goal is to "modulate" its internal features using the data from the Projection Module.
  • Loss Functions: The model is trained using a combination of:
  • Adversarial Loss: To ensure the output looks like a real face.
  • Perceptual Loss (VGG19): To maintain structural consistency.
  • Identity Preserving Loss (LightCNN): To ensure the synthesized person actually looks like the person in the thermal image.

Datasets

The study utilizes two primary datasets, modified with simulated atmospheric turbulence (random blur and elastic deformation):

  • VIS-TH: A challenging dataset containing paired Long Wave Infrared (LWIR) and visible images of 50 subjects with variations in expression and pose.
  • ARL-VTF: A large-scale thermal-to-visible face verification dataset with 220 identities.

The authors simulated turbulence based on physical models corresponding to a 300-meter long-range distance.

Key Insights and Ideas

  • End-to-End > Two-Stage: The most significant insight is that performing restoration and translation simultaneously is superior to sequential processing. This prevents the "compounding error" problem where a sub-optimal turbulence removal step limits the quality of the final visible image.
  • Generative Priors as a Safety Net: By using a pre-trained StyleGAN2, the network doesn't have to "learn" what a face looks like from scratch. It only has to learn how to map the messy thermal data onto the existing "manifold" of clean, visible faces.
  • Multi-Scale Modulation: Simply feeding a single latent vector into a GAN isn't enough for distorted images. By modulating the GAN at every resolution level (from $4 \times 4$ to $1024 \times 1024$), the model can correct both large-scale geometric warps and fine-scale blurs.

Key Results

The proposed method consistently outperformed baselines like HifaceGAN (One-Stage) and TDRN + HifaceGAN (Two-Stage):

  • Face Verification (VIS-TH): The method achieved a Rank-1 accuracy of 48.10%, a massive jump compared to the LightCNN baseline (18.10%) and the Two-Stage approach (32.38%).
  • Image Quality (ARL-VTF): On the ARL-VTF dataset, the method achieved the best scores across almost all metrics, including LPIPS (0.2185) and PSNR (19.06), indicating much higher visual fidelity and structural similarity to the ground truth.
  • Identity Preservation: The "Deg." metric (cosine distance in feature space) showed that the proposed method was significantly better at keeping the subject recognizable compared to "Direct" translation methods.

Limitations and Future Work

  • Simulation vs. Reality: While the authors used a sophisticated turbulence simulation, the model was primarily trained and tested on synthetic data. Real-world turbulence can be more unpredictable.
  • Computational Cost: Using StyleGAN2 as a backbone provides high quality but is computationally expensive compared to smaller, lighter translation networks.
  • Future Directions: The authors suggest that further investigation into network backbones and more diverse data augmentation strategies for real-world scenarios would be the next logical steps for this research.

Summary updated

Abstract

In many practical applications of long-range imaging such as biometrics and surveillance, thermal imagining modalities are often used to capture images in low-light and nighttime conditions. However, such imaging systems often suffer from atmospheric turbulence, which introduces severe blur and deformation artifacts to the captured images. Such an issue is unavoidable in long-range imaging and significantly decreases the face verification accuracy. In this paper, we first investigate the problem with a turbulence simulation method on real-world thermal images. An end-to-end reconstruction method is then proposed which can directly transform thermal images into visible-spectrum images by utilizing natural image priors based on a pre-trained StyleGAN2 network. Compared with the existing two-steps methods of consecutive turbulence mitigation and thermal to visible image translation, our method is demonstrated to be effective in terms of both the visual quality of the reconstructed results and face verification accuracy. Moreover, to the best of our knowledge, this is the first work that studies the problem of thermal to visible image translation under atmospheric turbulence.
BibTeX
@inproceedings{mei2022thermal,
  author = {Kangfu Mei and Yiqun Mei and Vishal M. Patel},
  title = {{Thermal to Visible Image Synthesis Under Atmospheric Turbulence}},
  booktitle = {ICIP},
  year = {2022},
  doi = {10.1109/icip46576.2022.9897975},
}

Loading related content...

Figure