TANGO: Real-World Unregistered RGB-Guided Hyperspectral Super-Resolution with Two-Camera Acquisitions

Matteo Kolyszko, Marco Buzzelli, Simone Bianco, Raimondo Schettini

TL;DR

TANGO introduces the first large-scale, real-world dataset (DUET) and a novel architecture for hyperspectral image super-resolution that doesn't require pre-registered inputs. By combining global optical flow, multiscale local alignment, and a learned gating mechanism, TANGO achieves state-of-the-art results while being significantly more computationally efficient than previous methods.

What Problem Does This Paper Address?

Hyperspectral imaging (HSI) is a powerful tool that captures dozens or hundreds of narrow spectral bands, allowing us to identify materials and scene properties invisible to the naked eye. However, hardware constraints force a trade-off: you can have high spectral resolution or high spatial resolution, but rarely both.

To fix this, researchers use RGB-guided super-resolution, where a high-resolution (HR) RGB image helps "sharpen" a low-resolution (LR) hyperspectral cube. The catch? Most existing algorithms assume the RGB and HSI images are perfectly pixel-aligned. In the real world, if you mount two different cameras side-by-side, they will have different viewpoints, focal lengths, and—most importantly—parallax. This geometric misalignment causes standard fusion methods to hallucinate artifacts or blur details.

Existing benchmarks often sidestep this by using synthetic data or very small datasets. This paper tackles the "messy reality" of dual-camera acquisitions head-on.

Methods

The authors propose TANGO (Two-camera AligNment with Gated Occlusions), a framework designed to handle unregistered inputs through a four-stage pipeline:

  1. Global Alignment: Since the cameras are physically separated, there is a dominant displacement. TANGO uses a fine-tuned RAFT (Recurrent All-Pairs Field Transforms) model to estimate a dense optical flow between a "pseudo-RGB" version of the HSI and the actual RGB reference.
  2. Multiscale Embedding: Both images are projected into a feature space at three different spatial resolutions. This allows the model to handle large displacements at coarse scales and fine details at higher resolutions.
  3. Local Alignment with Learned Gating: Even after global warping, local inconsistencies remain due to depth-dependent parallax and occlusions. TANGO uses local cross-attention to refine these matches. Crucially, it includes a learned gating mechanism that predicts how reliable the RGB guidance is at any given pixel. If a region is occluded in one view, the gate "shuts," preventing the model from injecting misleading information.
  4. Multiscale Spatial-Spectral Fusion: Finally, the aligned and gated features are fused using a transformer-based backbone to reconstruct the final high-resolution HSI.
Figure 1. Overview of the proposed framework. (a) We illustrate the dual-camera acquisition setup used to collect DUET, where an RGB camera and a hyperspectral camera are mounted side by side to capture 403 real HSI–RGB scene pairs under natural cross- camera misalignment. (b) We propose TANGO, a model for un- registered RGB-guided hyperspectral super-resolution that recon- structs a high-resolution HSI from a low-resolution HSI and an unregistered high-resolution RGB image.

Datasets

The paper introduces DUET, the first large-scale public benchmark for this task based on real hardware.

  • DUET: 403 indoor and outdoor scenes captured with a Sony $\alpha$6100 RGB camera and a Specim IQ hyperspectral camera mounted 15 cm apart. It includes diverse lighting, materials, and significant parallax.
  • REAL: A secondary benchmark with 57 scenes. While it features unregistered viewpoints, its RGB guidance is synthesized from HSI data, meaning it lacks the cross-sensor appearance variations found in DUET.
The physical dual-camera rig used to collect the DUET dataset.

Key Insights and Ideas

  • Parallax is not a Homography: The authors demonstrate that a single global transformation (like a homography) cannot register these images because the displacement depends on the depth of objects in the scene. This motivates their dense, flow-based approach.
  • The Power of Gating: One of the "aha" moments is the ablation study on the gating mechanism. Without it, the model tries to force-match pixels in occluded areas, which actually degrades performance. The gate allows the model to fall back on the HSI's own spectral correlations when the RGB guidance is untrustworthy.
  • Efficiency Matters: By using a tile-based processing approach and a more efficient fusion backbone, TANGO manages to outperform heavy competitors while using 2.5$\times$ fewer FLOPs.

Key Results

TANGO sets a new bar for performance across multiple metrics (PSNR, SSIM, and SAM):

  • On DUET: At a $\times$4 scale factor, TANGO achieves 43.67 dB PSNR, outperforming the strongest competitor (SSC-HSR) by 2.23 dB. The lead persists at $\times$8 and $\times$16 scales.
  • On REAL: It maintains its lead, improving PSNR by nearly 1 dB over the previous state-of-the-art (LRTN).
  • Qualitative Improvement: Visual results show that TANGO reconstructs sharp text and fine architectural details that other methods either blur or "double-image" due to misalignment.
Qualitative comparison showing TANGO's superior reconstruction of fine text and boundaries.

Limitations and Future Work

The primary limitation stems from the hardware used for the dataset: the Specim IQ is a scanning camera. This means the scene and illumination must remain perfectly static during the acquisition (which takes several seconds). Consequently, the DUET dataset does not include moving objects like cars or pedestrians.

Future work could involve extending these alignment techniques to video HSI-SR or developing methods that can handle dynamic scenes captured with snapshot hyperspectral sensors.

Summary updated

Abstract

Unregistered hyperspectral image super-resolution aims to recover a high-resolution HSI from a low-resolution hyperspectral observation and an unregistered high-resolution RGB reference. Although recent methods have begun to address the geometric misalignment between the two modalities, their evaluation is still constrained by the lack of a large-scale public benchmark based on real two-camera RGB--HSI acquisitions with a high-resolution HSI reference. To address this gap, we introduce DUET, the first large-scale public dataset for unregistered RGB-guided hyperspectral super-resolution, comprising 403 indoor and outdoor scenes captured with two separate RGB and hyperspectral cameras. We further propose TANGO (Two-camera AligNment with Gated Occlusions), which first compensates for the dominant cross-camera displacement through end-to-end global alignment and then refines the remaining inconsistencies with a coarse-to-fine local feature alignment. A learned gating mechanism further attenuates the RGB guidance where the two views cannot be consistently matched. Experimental results show that TANGO achieves state-of-the-art performance on both DUET and REAL across all evaluated scale factors, outperforming the strongest competitor on DUET by up to 2.29 dB in PSNR while requiring 2.5$\times$ fewer FLOPs. The code and dataset will be available at https://github.com/FagiolottoConSombrero/TANGO-DUET
BibTeX
@misc{kolyszko2026tango,
  author = {Matteo Kolyszko and Marco Buzzelli and Simone Bianco and Raimondo Schettini},
  title = {{TANGO: Real-World Unregistered RGB-Guided Hyperspectral Super-Resolution with Two-Camera Acquisitions}},
  year = {2026},
  eprint = {2611.22699},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22699},
}
Figure