TPTransformer: Tensor–Tensor Product Transformer for Hyperspectral Image Super-Resolution

Honghui Xu, Chuangjie Fang, Yiqun Meng, Jiawei Jiang, Sixian Chan, Shiqing Zhang, Jianwei ZhengCVPR 2026 Findings 2026 · OpenReview

TL;DR

TPTransformer is a novel deep unfolding network for hyperspectral image super-resolution that treats images as true 3D tensors rather than flattened matrices. By combining a learnable, data-adaptive $\ell_p$-norm for noise separation with a Fourier-accelerated Tensor-Tensor Product attention mechanism, it captures complex spatial-spectral correlations with near-linear computational complexity. The result is a highly interpretable framework that achieves state-of-the-art fusion performance across multiple benchmarks.

What Problem Does This Paper Address?

Hyperspectral imaging (HSI) captures hundreds of narrow, contiguous spectral bands, providing rich chemical and physical insights for applications like environmental monitoring and material identification. However, physical sensor constraints force a trade-off: you can have high spectral resolution (many bands) or high spatial resolution (sharp details), but rarely both at once. Hyperspectral image super-resolution (HSISR) solves this computationally by fusing a low-resolution hyperspectral image (LRHS) with a high-resolution multispectral image (HRMS).

Current state-of-the-art methods use Deep Unfolding Networks (DUNs) based on Robust Principal Component Analysis (RPCA) to split the image into a low-rank structural component and a sparse noise/distortion component. However, these models suffer from two major limitations:

  1. Rigid Sparsity Priors: They rely on a static $\ell_1$-norm prior to enforce sparsity, which fails to adapt to the diverse, heterogeneous noise distributions across different spectral bands.
  2. Loss of 3D Structure: Their low-rank branches use standard matrix-based Transformer attention, which flattens the spectral dimensions into 2D vectors. This discards the multiway spatial-spectral coupling and cyclic continuity inherent to 3D tensor data.

Methods

The authors propose an Adaptive $\ell_p$-RPCA Architecture that tackles both limitations simultaneously through two main innovations:

  • Adaptive Sparse Module: Instead of a fixed $\ell_1$-norm, the framework uses an adaptive $\ell_p$-norm regularization ($0 < p \le 1$). Crucially, the exponent $p$ is smoothly differentiable and learned dynamically via backpropagation. This allows the network to automatically adjust the sparsity constraint based on the local statistical characteristics of each band, balancing noise suppression and structural preservation.
  • Tensor-Tensor Product Transformer (TPTransformer): In the low-rank branch, the authors replace standard dot-product attention with a multilinear tensor interaction based on the t-product. Instead of projecting tokens into 1D vectors, features are projected into tube-shaped tensor segments. The attention mechanism computes circular convolutions along the spectral (tube) mode. To make this computationally efficient, the t-product is computed by transforming the tensors into the frequency domain using Discrete Fourier Transforms (DFTs), performing slice-wise matrix multiplications, and applying an inverse DFT. This reduces the computational complexity from quadratic to near-linear while preserving the integrity of the 3D tensor structure.
Figure 3. Adaptive p-RPCA overview.

Datasets

The model was trained and evaluated on three widely recognized hyperspectral benchmarks:

  • Cave: An indoor dataset rich in diverse textures and materials.
  • Pavia Center: An outdoor urban dataset captured via airborne sensors.
  • Chikusei: A large-scale agricultural and urban airborne dataset.

Downsampling scale factors of $\times4$ and $\times8$ were simulated using Gaussian blurring and spatial downsampling to evaluate super-resolution performance.

Key Insights and Ideas

  • The "Aha" Moment for Attention: Standard attention flattens 3D images into 2D matrices, meaning the model only measures second-order correlations and loses track of spectral continuity. By using the t-product in the Fourier domain, the attention mechanism treats each spectral tube as a circular convolution kernel, preserving multiway coherence.
  • Expressivity Gap: The authors mathematically prove a "Headwise Expressivity Gap"—a single t-product attention head strictly generalizes a standard dot-product head by generating multiple independent slice-wise attention patterns simultaneously under equal width constraints.
  • Why Learn $p$? Residual analysis of real hyperspectral data reveals a super-Gaussian distribution (sharp peaks and heavy tails) that varies significantly across bands. A fixed $p$ or $\ell_1$ norm is mathematically suboptimal. Learning $p$ dynamically allows the network to perform magnitude-adaptive shrinkage.
Matrix vs Tensor-Tensor Product Rank

Key Results

  • Quantitative Superiority: TPTransformer consistently outperformed six recent state-of-the-art methods (including diffusion-based and invertible networks) across all datasets. For instance, on the Chikusei dataset ($\times4$), it achieved a PSNR of 44.8661 dB (outperforming the next best by over 0.31 dB) and the highest SSIM (0.9702). On the Cave dataset ($\times8$), it achieved a massive PSNR of 51.3810 dB.
  • Efficiency: The model sits firmly on the Pareto frontier, delivering the highest reconstruction accuracy with moderate parameter counts (~10M at 3 stages) and competitive inference runtimes compared to heavy diffusion models.
  • Qualitative Fidelity: Visual error maps show that the proposed method yields significantly lower and more uniform residuals, successfully reconstructing sharp building edges, road textures, and thin specular highlights without the blur artifacts common in matrix-based methods.

Limitations and Future Work

  • While the Fourier-domain t-product reduces computational complexity to near-linear relative to token length, the model still relies on a fixed number of unfolding stages (the authors settle on 3). Scaling to even larger scene sizes or real-time video hyperspectral imaging might require further optimization of the stage-wise architecture.
  • The current degradation model assumes a known or simulated spatial/spectral response matrix; adapting the framework to completely blind HSISR where the degradation operators are entirely unknown remains an open avenue for future research.

Summary updated

Abstract

Hyperspectral image super-resolution (HSISR) aims to reconstruct a high-resolution hyperspectral image (HRHS) by fusing low-resolution hyperspectral (LRHS) and high-resolution multispectral (HRMS) observations. Deep unfolding networks (DUNs) have emerged as an effective framework for HSISR, bridging model-driven interpretability and data-driven efficiency. However, existing DUNs rely on static priors and matrix-based attention, limiting their ability to capture spectral-spatial correlations. To address these challenges, we propose a unified robust principal component analysis (RPCA)-based fusion framework that simultaneously enhances sparsity modeling and low-rank representation. In the sparse branch, an adaptive ℓ_p-norm regularization (0 < p ≤ 1) dynamically adjusts the sparsity constraint to the data distribution, yielding more flexible and accurate noise--structure separation. In the low-rank branch, we propose a Tensor--Tensor Product Transformer, which reformulates attention as a multilinear tensor interaction implemented via Discrete Fourier transforms. Specifically, the t-product is computed by transforming tensors into the frequency domain for efficient circular convolution and then inversely transformed back to the spatial domain. This tensor-wise attention mechanism captures cyclic spectral dependencies and spatial-spectral coherence while reducing computational complexity from quadratic to near-linear. Extensive experiments on multiple hyperspectral datasets demonstrate that the proposed framework achieves state-of-the-art fusion performance, with superior scalability and physical interpretability.
BibTeX
@inproceedings{xu2026tptransformer,
  author = {Honghui Xu and Chuangjie Fang and Yiqun Meng and Jiawei Jiang and Sixian Chan and Shiqing Zhang and Jianwei Zheng},
  title = {{TPTransformer: Tensor–Tensor Product Transformer for Hyperspectral Image Super-Resolution}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=vwIJ0D5PJK},
}

Loading related content...

Figure