Abstract
Planetary rovers often face mission-critical decisions based on terrain traversability, where distinguishing between solid rock and soft sand is vital. While thermal infrared (IR) imaging provides superior cues due to material-specific thermal inertia, many rovers lack dedicated IR sensors. This paper proposes MU-Net (Multiple U-Net), a deep learning framework that estimates thermal IR images from standard RGB data. By incorporating terrain classification as a joint task, MU-Net improves estimation accuracy by approximately 14% over baseline models, providing a "virtual" thermal sensor for resource-constrained robotic platforms.
Introduction
Navigating the Martian surface requires precise terrain classification. Standard RGB cameras can be ambiguous; for instance, dust-covered rocks may look similar to sand. Thermal IR data resolves this by capturing thermal inertia—the rate at which materials heat up and cool down. However, IR cameras add weight, power consumption, and complexity. This research explores whether cross-domain mapping can synthesize thermal data from RGB images, effectively giving rovers "thermal vision" without the hardware.
Methodology: The MU-Net Architecture
The proposed MU-Net architecture builds upon the U-Net model, traditionally used for medical segmentation. The authors introduce a multi-task learning approach where the network simultaneously predicts the thermal IR map and a terrain annotation map.
Two primary configurations were tested:
- MU-Net1 (Implicit Integration): A multi-task model where the encoder (contracting path) is shared. MU-Net1-b, which utilizes separate decoders (expansive paths) for thermal and annotation outputs, proved most effective.
- MU-Net2 (Explicit Integration): A sequential approach where features from a pre-trained classification network are concatenated into the IR estimation pipeline.
The underlying logic relies on the energy balance equation, where reflected solar energy (RGB) and emitted thermal energy (IR) are coupled through the physical properties of the terrain.
Experimental Setup and Dataset
Data was collected at the JPL Mars Yard using a FLIR Grasshopper RGB camera and a FLIR AX65 Thermal camera. The dataset consists of 416 registered image pairs captured over an 8-hour window to account for varying solar positions. Six terrain categories were labeled: soil, sand, rocks, bedrocks, rocky terrain, and ballast.
Results and Discussion
MU-Net1-b achieved a Mean Absolute Error (MAE) of 0.59 on normalized IR values, a significant improvement over the 0.69 MAE of the standard U-Net baseline.
Qualitatively, MU-Net produces smoother, more physically consistent thermal maps. It successfully identifies that sandy regions possess lower thermal inertia (appearing cooler) compared to rocks, even when visual textures are subtle. Furthermore, the model demonstrated robustness across different times of day, maintaining performance even when tested on lighting conditions not seen during specific training segments.
Conclusion
MU-Net demonstrates that terrain classification is a powerful constraint for cross-domain image estimation. By teaching the model what it is seeing, it better predicts how that material responds thermally. Future work will aim to incorporate environmental variables such as local solar incidence angles and shadows to further refine the "virtual" thermal sensor for deployment in rugged, unmapped planetary environments.
Summary updated