Cross-Modal Guidance for Spectral Image Super-Resolution

20 concepts14 papers2 proposals

In many imaging scenarios, there is a fundamental trade-off between spatial resolution and spectral richness. Standard RGB cameras capture high-frequency spatial details but only three broad color bands. Conversely, multispectral and thermal sensors capture valuable information about material composition or heat signatures but are often limited by hardware constraints to much lower resolutions.

Cross-modal guidance addresses this gap by using a high-resolution "guide" image—typically a standard color (RGB) or panchromatic image—to sharpen a low-resolution "target" image from a different modality.

The Structural Blueprint

The core intuition behind guided super-resolution is that while different sensors capture different physical properties, they often share the same underlying scene geometry. An edge in a thermal image, representing a temperature transition, often aligns with a visible edge in an RGB image, representing a change in material or lighting.

Mehrdad Shoeiby et al. demonstrated this in their work on the StereoMSI dataset, showing that incorporating high-resolution RGB guidance significantly outperforms using spectral data alone. By treating the RGB image as a spatial guide, models can transfer high-frequency textures and structural information to information-rich but pixel-poor multispectral data.

This process is often framed as "guided upsampling." As Wenzhi Liao et al. argue in their study of multiresolution thermal and color data, this approach allows researchers to maintain sharp edges in thermal maps while preserving the "spectral integrity" of the original sensor. The high-resolution visible imagery acts as a blueprint, ensuring that the upsampled data respects the physical boundaries of objects in the scene.

Aligning Gradients Across Modalities

A major challenge in this field is that the relationship between modalities is not always linear. A bright object in the visible spectrum might be cold (dark) in the thermal spectrum. To handle this, researchers focus on aligning spatial gradients rather than raw pixel values.

Yvette Y. Lin et al. utilize this concept in ThermalNeRF by employing cross-spectral regularization. By aligning the gradients of visible and thermal radiance fields, the authors can synthesize thermal reconstructions that surpass the native resolution limits of the hardware. Similarly, Hang Zhong et al. propose SwinPaste, which uses attention mechanisms to selectively "paste" high-frequency edges from the visible spectrum onto thermal heat maps, effectively filling in details that are physically blurred in the infrared domain.

From Satellite Imagery to Medical Screening

The applications of this technique span from the planetary scale to human health. In remote sensing, Muhammad Sarmad et al. developed DiffFuSR, a diffusion-based model that super-resolves Sentinel-2 satellite bands. By extracting spatial priors from high-resolution visible bands, the authors can sharpen lower-resolution multispectral bands without distorting their physical measurements.

On a smaller scale, Jing Wei Chin et al. show how this fusion enables remote body temperature screening. By using high-resolution RGB imagery to localize anatomical targets like the forehead, the system can extract accurate temperatures from indistinct heat signatures that would otherwise be too small for the thermal sensor to resolve at a distance.

Bidirectional and Generative Refinement

While most work focuses on RGB guiding other sensors, the relationship can be bidirectional. Dongyuan Li et al. introduced ThermNet, which uses thermal maps to guide the restoration of noisy RGB images in low-light environments. Because thermal sensors are unaffected by darkness, they provide a stable structural reference that prevents the blurring and color shifting common in purely visible-light enhancement.

Recent advances have moved toward more sophisticated architectures to handle the complexity of cross-modal fusion. Jiyong Rao notes in the PBVS 2025 challenge results that Transformer and Mamba-based models are now achieving record-breaking accuracy in high-factor upscaling (up to ×16). These models, such as the lightweight SwinFuSR proposed by Cyprien Arnold et al., leverage cross-domain guidance to reconstruct textures that are entirely unrecoverable using single-modality methods.

Key Foundational Work

PIRM2018 Challenge on Spectral Image Super-Resolution: Dataset and Study (2019) — contributed 2 concept elements to this direction.
Processing of Multiresolution Thermal Hyperspectral and Digital Color Data: Outcome of the 2014 IEEE GRSS Data Fusion Contest (2015) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_color_guided_super_resolution_cfc9773c,
  title = {color-guided super-resolution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/color-guided-super-resolution/rdr_cfc9773ca2304b42b56b55d99ddb0508},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_color_guided_super_resolution_cfc9773c_v1,
  title = {color-guided super-resolution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/color-guided-super-resolution/rdr_cfc9773ca2304b42b56b55d99ddb0508/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake