Abstract
There are two mast cameras (Mastcam) onboard the Mars rover Curiosity. Both Mastcams are multispectral imagers with nine bands in each. The right Mastcam has three times higher resolution than the left. In this chapter, we apply some recently developed deep neural network models to enhance the left Mastcam images with help from the right Mastcam images. Actual Mastcam images were used to demonstrate the performance of the proposed algorithms.
Introduction
The Curiosity rover's imaging capabilities are anchored by its two Mastcams. While both are multispectral, they possess different optical characteristics: the right Mastcam features a 100mm focal length, providing triple the spatial resolution of the 34mm left Mastcam. Conversely, the left camera offers a significantly wider field of view.
Historically, creating 3D stereo images or fused 12-band "image cubes" required downsampling the high-resolution right images to match the lower-resolution left ones. This process discards high-frequency spatial data, limiting the quality of visuals for scientific analysis and public engagement. The objective of this research is to perform Super-Resolution (SR) on the left images using the right camera's data as a guide, while strictly maintaining spectral integrity to ensure accurate mineralogical representation.
Methodology: Unsupervised Sparse Dirichlet Network (uSDN)
The core of the proposed enhancement framework is the unsupervised sparse Dirichlet Network (uSDN). Unlike traditional "black box" deep learning models, the uSDN incorporates physical constraints through a Dirichlet distribution.
Key Architectural Features:
- Physical Constraints: The network treats each pixel as a linear combination of spectral bases. The Dirichlet constraint ensures these combinations are non-negative and sum to one, reflecting the physical reality of material mixing in Martian soil and rock.
- Weight Sharing and Robust Loss: Encoder weights are shared between the high-resolution and low-resolution networks to improve consistency. The implementation utilizes an $l_{21}$ loss function, which is more robust than standard $l_2$ loss for preserving spectral information across pixels.
- Hybrid Dirichlet-Net + U-Net: To address mis-registration between the two cameras, the authors integrated a U-Net architecture. This allows the system to "switch" low-resolution spatial patches with high-resolution counterparts, effectively bypassing alignment errors.
- Transition Learning: A dedicated module learns relationships between neighboring pixels to ensure smooth gradients and adds high-frequency residuals, preventing "patchy" artifacts in the final output.
Datasets and Validation
The models were initially validated using the CAVE and Harvard datasets, standard benchmarks for hyperspectral and multispectral image fusion. For the primary Martian application, actual Curiosity rover images (e.g., test image MSL_0002_0114_M1) were used. In this setup, the right camera data served as the High-Resolution Multispectral Image (HR MSI) and the left as the Low-Resolution Hyperspectral Image (LR HSI).
Results and Discussion
The uSDN approach demonstrated clear quantitative and qualitative superiority over existing methods like Coupled Non-negative Matrix Factorization (CNMF).
- Quantitative Performance: The uSDN achieved a Root Mean Squared Error (RMSE) of 0.033, significantly lower than CNMF's 0.056. The Spectral Angle Mapper (SAM) score was 2.09 (vs. 2.48), indicating minimal spectral distortion.
- Visual Quality: Qualitative analysis confirmed that the proposed method produces sharper images than standard bicubic interpolation or deep learning models trained on terrestrial datasets (like EnhanceNet).
- Registration Resilience: The hybrid architecture successfully fused images even when the left and right camera views did not perfectly overlap.
Conclusion and Future Work
While the current model effectively enhances Martian imagery, computational requirements necessitate processing small image patches, which can occasionally lead to boundary artifacts. Future research will focus on refining transition learning to eliminate these artifacts and scaling the system to process larger image sets from the Curiosity mission more efficiently. This work paves the way for higher-fidelity VR/AR experiences and more precise scientific analysis of the Martian surface.
Summary updated