T2V-DDPM: Thermal to Visible Face Translation using Denoising Diffusion Probabilistic Models

Nithin Gopalakrishnan Nair, Vishal M. PatelarXiv 2022 · OpenReview

TL;DR

This paper introduces T2V-DDPM, a novel approach for translating thermal infrared face images into their visible spectrum counterparts using Denoising Diffusion Probabilistic Models (DDPMs). They also propose an accelerated inference strategy for DDPMs, achieving state-of-the-art results in thermal-to-visible face translation with improved efficiency.

What Problem Does This Paper Address?

Modern surveillance systems rely heavily on facial recognition, which typically uses visible spectrum images. However, visible light cameras struggle in low-light or nighttime conditions, making thermal infrared cameras a valuable alternative. The challenge lies in the significant domain gap between thermal and visible images, which hinders direct facial recognition. A common solution is to translate thermal images into the visible domain, a task known as Thermal-to-Visible (T2V) image translation. While Generative Adversarial Networks (GANs) have been used for this, they can be difficult to train and prone to issues like mode collapse. This paper aims to address the T2V translation problem with a more robust and high-quality generative model, DDPMs, and to overcome the inherent slow inference speed of DDPMs.

Methods

The core of the proposed method is the application of Denoising Diffusion Probabilistic Models (DDPMs) for T2V face translation. DDPMs work by learning to reverse a diffusion process that gradually adds noise to an image. During training, the model learns the conditional distribution of visible facial images given their corresponding thermal images. The thermal image acts as a condition, guiding the denoising process.

A key contribution is a novel inference strategy to speed up DDPMs. The authors observed that initial diffusion steps primarily learn coarse image features like background color and overall shape. Instead of starting the inference from pure Gaussian noise, they propose starting with a "coarse image" derived from the thermal image. This coarse image is generated by creating a mask from the thermal image and combining it with a sampled background color. By skipping the initial noise-adding steps and starting the denoising process from a more informed state (closer to the target image), they significantly reduce the number of required inference steps, leading to up to a 2x speedup without performance degradation.

Datasets

The paper evaluates their method on three datasets:

  • VIS-TH dataset: Contains aligned facial images from LWIR modality for 50 identities.
  • ARL-VTF dataset: Also features LWIR facial images, with visible images that are often overexposed. The authors correct this overexposure through exposure matching.
  • A challenging dataset (implied to be VIS-TH based on Figure 4): Used to demonstrate generalization capabilities with a low number of training image pairs.

The authors use a train-test split strategy similar to previous works [16], [8] for these datasets.

Key Insights and Ideas

  • DDPMs are well-suited for T2V translation: The paper demonstrates that DDPMs, with their ability to model complex conditional distributions, are a powerful tool for translating thermal to visible face images, achieving high-quality results.
  • Early diffusion steps capture coarse features: A crucial observation is that the initial stages of the DDPM diffusion process are responsible for learning low-level semantics and coarse image structures. This insight is the foundation for their accelerated inference method.
  • Starting inference from a "coarse image" is effective: By leveraging the thermal image to create an initial "coarse" visible image, the DDPM can skip many early denoising steps, significantly speeding up inference without sacrificing quality. This is particularly relevant for real-time applications.
  • Generalization with limited data: The proposed method shows good generalization capabilities even with a low number of training image pairs, outperforming existing GAN-based methods in this regard.

Key Results

  • State-of-the-art performance: The T2V-DDPM method achieves state-of-the-art results on multiple datasets for T2V face translation.
  • Improved facial verification accuracy: On the VIS-TH dataset, the method achieves a Rank-1 accuracy of 70.5%, an improvement of 2% over the best previous method. For VR@FAR=0.1%, they achieve 23.5% (an 8.3% improvement), and for VR@FAR=1%, they achieve 45.5% (a 6.3% improvement). Similar gains are observed on the ARL-VTF dataset.
  • Accelerated inference: The novel inference strategy provides up to a 2x speedup in inference time without any drop in performance.
  • High-quality image generation: Qualitative results (Figures 4 and 5) show that the generated visible images are semantically similar to the thermal inputs and possess realistic facial details, even handling occlusions like hats and sunglasses better than some GAN-based methods.

Limitations and Future Work

  • Inference Speed: While their proposed method significantly speeds up DDPM inference, it is still noted that DDPM inference is generally slower than conventional CNNs due to its iterative nature.
  • Future Work: The paper does not explicitly detail future work directions, but the success in accelerating DDPM inference for T2V translation suggests potential applications in other conditional image generation tasks where inference speed is critical. Further exploration of different coarse image generation strategies or more advanced DDPM architectures could also be avenues for future research.

Summary updated

Abstract

Modern-day surveillance systems perform person recognition using deep learning-based face verification networks. Most state-of-the-art facial verification systems are trained using visible spectrum images. But, acquiring images in the visible spectrum is impractical in scenarios of low-light and nighttime conditions, and often images are captured in an alternate domain such as the thermal infrared domain. Facial verification in thermal images is often performed after retrieving the corresponding visible domain images. This is a well-established problem often known as the Thermal-to-Visible (T2V) image translation. In this paper, we propose a Denoising Diffusion Probabilistic Model (DDPM) based solution for T2V translation specifically for facial images. During training, the model learns the conditional distribution of visible facial images given their corresponding thermal image through the diffusion process. During inference, the visible domain image is obtained by starting from Gaussian noise and performing denoising repeatedly. The existing inference process for DDPMs is stochastic and time-consuming. Hence, we propose a novel inference strategy for speeding up the inference time of DDPMs, specifically for the problem of T2V image translation. We achieve the state-of-the-art results on multiple datasets. The code and pretrained models are publically available at http://github.com/Nithin-GK/T2V-DDPM
BibTeX
@misc{nair2022t,
  author = {Nithin Gopalakrishnan Nair and Vishal M. Patel},
  title = {{T2V-DDPM: Thermal to Visible Face Translation using Denoising Diffusion Probabilistic Models}},
  year = {2022},
  eprint = {2209.08814},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.08814},
}
Figure