MU-Net: Deep Learning-Based Thermal IR Image Estimation From RGB Image

Yumi Iwashita, Kazuto Nakashima, Sir Rafol, Adrian Stoica, Ryo KurazumeWorkshops @ CVPR 2019 · DOI

Abstract

Planetary rovers often face mission-critical decisions based on terrain traversability, where distinguishing between solid rock and soft sand is vital. While thermal infrared (IR) imaging provides superior cues due to material-specific thermal inertia, many rovers lack dedicated IR sensors. This paper proposes MU-Net (Multiple U-Net), a deep learning framework that estimates thermal IR images from standard RGB data. By incorporating terrain classification as a joint task, MU-Net improves estimation accuracy by approximately 14% over baseline models, providing a "virtual" thermal sensor for resource-constrained robotic platforms.

Introduction

Navigating the Martian surface requires precise terrain classification. Standard RGB cameras can be ambiguous; for instance, dust-covered rocks may look similar to sand. Thermal IR data resolves this by capturing thermal inertia—the rate at which materials heat up and cool down. However, IR cameras add weight, power consumption, and complexity. This research explores whether cross-domain mapping can synthesize thermal data from RGB images, effectively giving rovers "thermal vision" without the hardware.

Methodology: The MU-Net Architecture

The proposed MU-Net architecture builds upon the U-Net model, traditionally used for medical segmentation. The authors introduce a multi-task learning approach where the network simultaneously predicts the thermal IR map and a terrain annotation map.

The MU-Net architecture features a shared contracting path and specialized expansive paths for thermal estimation and terrain classification.

Two primary configurations were tested:

  1. MU-Net1 (Implicit Integration): A multi-task model where the encoder (contracting path) is shared. MU-Net1-b, which utilizes separate decoders (expansive paths) for thermal and annotation outputs, proved most effective.
  2. MU-Net2 (Explicit Integration): A sequential approach where features from a pre-trained classification network are concatenated into the IR estimation pipeline.

The underlying logic relies on the energy balance equation, where reflected solar energy (RGB) and emitted thermal energy (IR) are coupled through the physical properties of the terrain.

Experimental Setup and Dataset

Data was collected at the JPL Mars Yard using a FLIR Grasshopper RGB camera and a FLIR AX65 Thermal camera. The dataset consists of 416 registered image pairs captured over an 8-hour window to account for varying solar positions. Six terrain categories were labeled: soil, sand, rocks, bedrocks, rocky terrain, and ballast.

Results and Discussion

MU-Net1-b achieved a Mean Absolute Error (MAE) of 0.59 on normalized IR values, a significant improvement over the 0.69 MAE of the standard U-Net baseline.

Qualitative comparison: (a) Input RGB, (b) Ground Truth Thermal, (c) Baseline Estimation, and (d) MU-Net Estimation.

Qualitatively, MU-Net produces smoother, more physically consistent thermal maps. It successfully identifies that sandy regions possess lower thermal inertia (appearing cooler) compared to rocks, even when visual textures are subtle. Furthermore, the model demonstrated robustness across different times of day, maintaining performance even when tested on lighting conditions not seen during specific training segments.

Conclusion

MU-Net demonstrates that terrain classification is a powerful constraint for cross-domain image estimation. By teaching the model what it is seeing, it better predicts how that material responds thermally. Future work will aim to incorporate environmental variables such as local solar incidence angles and shadows to further refine the "virtual" thermal sensor for deployment in rugged, unmapped planetary environments.

Summary updated

BibTeX
@inproceedings{iwashita2019mu,
  author = {Yumi Iwashita and Kazuto Nakashima and Sir Rafol and Adrian Stoica and Ryo Kurazume},
  title = {{MU-Net: Deep Learning-Based Thermal IR Image Estimation From RGB Image}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00134},
}
Figure