TL;DR
This paper introduces PETIT-GAN, a novel generative adversarial network that leverages physical properties of thermal imaging to improve image-to-image translation. By fusing a physically-informed estimator with a deep learning model, PETIT-GAN significantly enhances the fidelity and robustness of translating between different thermal spectral bands, addressing the lack of high-resolution thermal multispectral datasets.
What Problem Does This Paper Address?
The field of remote sensing is increasingly benefiting from high-resolution imagery, particularly from nanosatellites. While visible and near-infrared (VNIR) spectrum data is abundant, there's a critical gap in publicly available, high-resolution thermal multispectral imagery (specifically in the longwave infrared, LWIR, spectrum). This lack of data hinders the development of advanced applications in areas like climate monitoring and fire detection.
Image-to-image (I2I) translation offers a potential solution by synthesizing thermal multispectral data from other visual modalities. However, a major challenge in I2I translation, especially for unpaired data (where corresponding images from different modalities are not perfectly aligned), is achieving high fidelity and robust training. Thermal imaging presents unique physical properties that, if exploited, could mitigate these challenges. This paper aims to bridge this gap by developing a physically enhanced I2I translation model for thermal imagery.
Methods
The core innovation of PETIT-GAN lies in its "physically enhanced" approach. Instead of relying solely on deep learning, it incorporates a physically-informed estimator.
- Physical Estimator: This component leverages the physics of thermal radiation. It models the relationship between an object's temperature, the camera's intrinsic temperature, and the resulting pixel intensity. This is based on Planck's law and the Stefan-Boltzmann law, extended to account for emissivity and camera characteristics. The authors developed a method to calibrate this estimator to predict monochromatic thermal intensity from panchromatic input and camera temperatures.
- Deep Estimator: Standard I2I translation architectures like CycleGAN and CUT are used as baselines. These typically employ encoder-decoder networks with residual blocks.
- PETIT-GAN Fusion: The proposed method fuses the physical estimator with a deep generative architecture. The physical estimator provides a prior approximation of the desired output, and a deep estimator learns the residual error. This fusion is expressed as: $G_{PETIT}(x) = G_{phys}(x) + G_{deep}(x)$. The authors also incorporate the camera's intrinsic temperature as an input to the deep estimator, allowing it to disentangle temperature-dependent transformations.
Datasets
The paper introduces a novel, in-house assembled aerial thermal multispectral dataset. This dataset was collected using a light airplane equipped with a FLIR Tau2 thermal camera. Images were captured with and without a 9µm IR bandpass filter, resulting in panchromatic (wide-band) and monochromatic (narrow-band) images, respectively. Crucially, due to the nature of aerial data collection, these datasets are unpaired, meaning images from different flights are not spatially registered.
Key Insights and Ideas
- Physics-Informed Deep Learning: The paper demonstrates that incorporating physical principles into deep learning models can significantly improve performance, especially in domains with well-understood underlying physics like thermal imaging.
- Bridging the Gap: The physical estimator acts as a powerful prior, guiding the deep learning model towards more accurate and realistic thermal translations, particularly when dealing with unpaired data.
- Robustness through Fusion: Fusing the physical estimator with the deep network not only improves accuracy but also leads to a more robust training procedure, potentially by guiding the model towards flatter minima in the loss landscape.
- Intrinsic Temperature as Input: Providing the camera's intrinsic temperature as an input to the deep network helps disentangle its effects from the inter-modal translation task, leading to better results.
Key Results
- Quantitative Improvement: PETIT-GAN achieved an approximate 50% improvement in mean Fréchet Inception Distance (FID) score compared to state-of-the-art baselines (CycleGAN and CUT).
- Improved Robustness: The standard deviation of FID scores was significantly reduced, indicating a more stable and reliable performance across different training runs.
- Qualitative Superiority: Visual comparisons show that PETIT-GAN produces outputs with greater fidelity and fewer spurious artifacts compared to baseline methods. The generated monochromatic images are more visually similar to real reference images.
- Physical Estimator's Impact: Configurations that included the physical estimator consistently outperformed those without it, highlighting its crucial role. The intrinsic temperature input was most beneficial when combined with the physical estimator.
Limitations and Future Work
- Calibration Accuracy: While the physical estimator is a key component, its accuracy is dependent on the calibration process. Improvements in calibration could further enhance PETIT-GAN's performance.
- Intrinsic Temperature Redundancy: The paper notes that the necessity of intrinsic temperature information when the physical estimator is absent requires further investigation.
- Single Modality Translation: The current method focuses on translating to a single monochromatic channel. Future work could extend this to multi-modal translation, synthesizing a complete multispectral dataset by repeating the process for different spectral bands or performing simultaneous multi-modal transformations.
Summary updated