Hyperspectral cameras record hundreds of narrow wavelength bands per pixel but at coarse spatial resolution. Multispectral cameras record a few broad bands at fine resolution. Fusing the two into one sharp, spectrally rich image is a standard remote-sensing task. The most principled way to do it models the unknown high-resolution cube as a small sum of "rank-1" pieces, each an outer product of a horizontal profile, a vertical profile, and a spectrum. This is a canonical polyadic decomposition (CPD), and it matches the physics of how materials mix in a pixel. Both sensors are then explained as blurred or band-averaged views of that one shared cube. These coupled methods have identifiability guarantees, but they treat every measured entry of both sensors as equally trustworthy. Real acquisitions are not like that: clouds cover one sensor's scene and not the other's, detectors die along one column, and whole bands saturate or drop out. The fit then reproduces the corruption in the recovered image.
This note proposes giving coupled tensor fusion a per-entry trust score that is recomputed as the fit proceeds, in the spirit of dynamic sensor weighting. The score is estimated from disagreement between the two sensors, not from each sensor's own fitting error. The score map is itself constrained to be a low-rank tensor, so that the common corruption shapes can be represented and the rest cannot. Those shapes are a region across all bands (a cloud), a band across all pixels (a dead band), and a column across some bands (a stripe). The central bet is simple: a sensor cannot reliably diagnose its own structured corruption, because a low-rank model will happily fit a low-rank artifact, but the other sensor can.
Core Claim
Coupled CPD fusion of hyperspectral and multispectral images degrades under structured, sensor-specific corruption. Reweighting the data fit with a low-rank, nonnegative reliability tensor, derived from cross-sensor inconsistency, should recover most of the gap to an oracle that knows where the corruption is. Self-residual robust reweighting and static common/distinct decompositions should fall short specifically when the corruption is itself low-rank. If self-residual reweighting matches the proposal on structured corruption, the claim is false, and the note says in advance how that would be detected.
Where This Sits
The tensor side supplies the method. Hyperspectral Super-Resolution: A Coupled Tensor Factorization Approach introduces STEREO, which fits one CPD whose degraded versions explain both sensors. It proves identifiability under mild rank conditions and reports a reconstruction signal-to-noise ratio (R-SNR) of 38.62 dB on SALINAS. It assumes uniform trust in every measurement, and that assumption is exactly what we relax.
Personalized Coupled Tensor Decomposition for Multimodal Data Fusion is the closest existing work. It splits each sensor into a shared tensor plus a sensor-specific "distinct" tensor that absorbs artifacts such as simulated clouds on Indian Pines (0 to 9.7% coverage). It also proves that the shared factors stay identifiable when one dataset is fully unique and the others are unique in only one mode. Its split between shared and distinct parts is governed by static regularizers, and the distinct part is itself a CPD. A corruption that the distinct CPD cannot represent therefore leaks into the shared image. A corruption that it can represent competes with genuine sensor-specific signal. We keep its identifiability result as the theoretical anchor and replace the static trade-off with entry-level trust.
Learning Tensor Low-Rank Prior for Hyperspectral Image Reconstruction shows that unrolling a tensor solver into a trained network cuts reconstruction from over 6,900 seconds to 0.36 seconds on ICVL. We borrow its unrolling recipe for the fast variant.
The dynamic-weighting side supplies the objective and the warnings. Provable Dynamic Fusion for Low-Quality Multimodal Data proves that dynamic fusion beats static fusion when weights are negatively correlated with each modality's error. Its method, QMF, reaches a weight–loss Pearson correlation of −0.735 on UPMC FOOD101. It operates on whole feature vectors of deep classifiers, and it relies on energy-based confidence that can be overconfident on unfamiliar noise. We use its principle as a measurable target at the level of individual tensor entries. We use a model-free cross-sensor signal instead of learned confidence.
EGM-Det and RGB-T Object Tracking: Benchmark and Baseline motivate the design. The first gates RGB versus infrared pixel by pixel. The second, the SGT tracker, reweights modalities by patch reconstruction error. Both show that reliability varies within a single scene. Neither exploits spatial–spectral multilinear structure. SGT's self-reconstruction error is also exactly the self-diagnosis signal we argue fails on structured corruption.
Interpretable Tensor Fusion assigns relevance weights to unimodal features and their outer products, but only once per dataset and only for one-dimensional feature vectors. It is the closest "tensor plus modality weights" work outside remote sensing. It differs from ours in being static and non-spatial.
Research Program
Let $\mathcal{Y}_h$ be the hyperspectral cube (coarse grid, full spectrum) and $\mathcal{Y}_m$ the multispectral cube (fine grid, few bands). Let $\mathcal{C} = [\![A, B, S]\!]$ be the rank-$R$ CPD of the unknown super-resolved cube, with spatial factors $A, B$ and spectral factor $S$. The known degradations are spatial blur-and-downsample matrices $P_1, P_2$ and the multispectral spectral-response matrix $P_3$. The weighted coupled objective is
$$\min_{A,B,S}\; \big\| \mathcal{W}_h \odot (\mathcal{Y}_h - \mathcal{C}\times_1 P_1 \times_2 P_2) \big\|_F^2 + \lambda \big\| \mathcal{W}_m \odot (\mathcal{Y}_m - \mathcal{C}\times_3 P_3) \big\|_F^2,$$
In this objective, $\odot$ is entrywise product, $\times_n$ is multiplication along mode $n$, $\lambda$ balances the two sensors, and $\mathcal{W}_h, \mathcal{W}_m \in [0,1]$ are reliability tensors of the same size as each measurement. Fixing the weights gives a weighted alternating least squares step. Alternating between weights and factors makes the fit dynamic.
The weights come from two signals. The first is a model-free consistency tensor, $\mathcal{R} = \mathcal{Y}_h \times_3 P_3 - \mathcal{Y}_m \times_1 P_1 \times_2 P_2$. It degrades each sensor into the other's common coarse, few-band space, where they must agree if both are clean. $\mathcal{R}$ says where the sensors disagree, but not which one is wrong.
The second signal handles attribution. At each flagged location, the current shared estimate $\mathcal{C}$ is compared with each sensor. The comparison is carried into the modality's own full grid by back-projecting $\mathcal{R}$ through the respective operator. The sensor whose residual carries the disagreement loses trust there.
The raw trust scores are then projected onto a nonnegative low-rank tensor, $\mathcal{W} \approx \mathbf{1} - \sum_{q=1}^{Q} a_q \circ b_q \circ s_q$, with small $Q$ (a handful of terms). This is the specifically tensor ingredient. Clouds, dead bands and stripes are each one or a few outer-product terms, so they are represented compactly and absorbed at little cost. Isolated noisy entries are not representable, so they cannot push the weights around and are left to the ordinary fit.
The research program has three parts. The first is the optimization-based method above. It keeps a distinct term from the personalized decomposition for genuinely sensor-specific signal, and adds a check that weights never fall below a floor across an entire mode, to avoid annihilating a factor.
The second is an analysis of identifiability under weighting. Zero weights turn the problem into CPD with missing entries. We will state conditions, adapted from the uni-mode uniqueness result of the personalized decomposition paper, under which the shared cube remains identifiable. An example: if the multispectral image stays unique on its clean region and the hyperspectral cube keeps spectral-mode uniqueness on its unweighted bands. We will verify these conditions empirically by factor-recovery error on synthetic cubes.
The third part is an unrolled, amortized version, following the approach of the TLPLN paper. Here a small network maps $\mathcal{R}$ and the current residuals to the low-rank weight factors at each of a fixed number of unrolled iterations. It is trained on synthetic corruptions and tested on held-out corruption types, which directly probes the overconfidence failure noted for learned reliability estimators.
The main risk is misregistration. The file notes that adaptive gates confuse spatial offsets between sensors with sensor noise, and a small shift makes $\mathcal{R}$ light up along every edge. We therefore include registration offset as a stress axis and report where the method breaks, rather than assuming calibrated operators.
Baselines
The naive reference point is STEREO from Hyperspectral Super-Resolution: A Coupled Tensor Factorization Approach with uniform weights. It is the exact model we modify, so any gain is attributable to the weights. Its reported clean-data R-SNR on SALINAS is 38.62 dB. We will reproduce that number first, and the proposal must not fall materially below it on clean data, since weights should stay near one when the sensors agree.
The strongest existing method is the personalized coupled decomposition of Personalized Coupled Tensor Decomposition for Multimodal Data Fusion. It is built precisely for sensor-specific artifacts such as clouds. We will use its alternating least squares solver, the same rank $R$ as our method, and a tuning grid over its regularizer and distinct-tensor rank of the same size as our grid over $Q$ and $\lambda$. For the corrupted setting its summary reports only that the solver "maintained low error at 20dB SNR", with no numerical value. We found no reported result on Lacuna for it on Indian Pines under cloud contamination, so we will run it ourselves. Beating it means higher R-SNR and lower spectral angle error on the recovered cube across corruption levels, with non-overlapping variability across random corruption draws.
The ablation that isolates the key ingredient is self-residual reweighting. It is the same weighted solver, but the weights are computed from each sensor's own fit residual by iteratively reweighted least squares with a Huber-style rule, in the manner of SGT's reconstruction-based reweighting. We also run a second ablation that keeps the cross-sensor signal but drops the low-rank constraint on $\mathcal{W}$, giving free per-entry weights. It tests whether the tensor structure of the reliability map matters.
A binary-masking baseline runs an off-the-shelf threshold on $\mathcal{R}$ to mark corrupted entries as missing, then applies STEREO. It mirrors the "drop the bad sensor" option at entry granularity.
Finally, an oracle STEREO that is given the true corruption mask bounds what any weighting can achieve. We report the fraction of the gap between naive STEREO and the oracle that each method closes. This normalizes across corruption strengths.
Why not just…?
Why not just use a robust loss (iteratively reweighted least squares or a sparse outlier term) on each sensor? This is the real threat, and the self-residual ablation exists to settle it. Our argument is that a rank-$R$ CPD can fit a rank-1 artifact such as a stripe or a dead band with one spare component. The artifact then produces a small self-residual and never gets down-weighted. The other sensor, which does not share the artifact, exposes it. We predict that self-residual reweighting matches the proposal on scattered impulse noise but loses clearly on stripes, dead bands and clouds. If it matches on structured corruption too, the cross-sensor idea adds nothing, and we will report that.
Why not just use the personalized common/distinct decomposition, which already absorbs clouds? It does absorb artifacts that its distinct CPD can represent. However, its trade-off is fixed by a static regularizer for the whole scene, and nothing tells it where the artifact is. It is our strongest baseline. The experiment that distinguishes the two varies corruption type and strength at a fixed regularizer. We expect the static split to need retuning per scene, while cross-sensor weights adapt without retuning.
Why not just detect corruption and mask it out? Masking is our binary baseline. Hard thresholds on $\mathcal{R}$ either miss faint haze or erase clean structure near edges and material boundaries, where the two sensors' different point-spread functions already disagree slightly. Continuous low-rank weights should degrade more gracefully. The comparison is scored as the fraction of the oracle gap closed.
Why not just a deep 2D fusion network with entropy gating, as EGM-Det does for RGB–infrared? Such gating works well for detection. EGM-Det reports 71.4% mAP50-95 on refined DroneVehicle and 60.3% on VEDAI. However, detection tolerates spectral distortion that reconstruction does not. A learned gate must also have seen the corruption type during training, which is exactly where the file notes learned reliability scores become overconfident. Our unrolled variant is the fair learned competitor, tested on held-out corruption types. We do not claim advantage on detection benchmarks.
Proposed Experiments
All experiments use SALINAS, Indian Pines, Pavia University and Cuprite. Each high-resolution cube is degraded into a hyperspectral–multispectral pair with known operators, as in the STEREO paper. We then inject four corruption families into one or both sensors: synthetic clouds at the 0–9.7% coverage used for Indian Pines in the personalized decomposition paper, column stripes on subsets of bands, whole-band dropout, and scattered impulse noise. Each has several strengths and random draws. We report R-SNR, spectral angle error, factor-recovery error on synthetic low-rank cubes, and the fraction of the oracle gap closed. The negative-correlation criterion of the QMF paper is also computed as the Pearson correlation between the learned weights and the true per-entry corruption magnitude. QMF's −0.735 on FOOD101 is a reference for scale only, not a comparable number.
The decisive check is the interaction between corruption type and weighting signal. The proposal should beat self-residual reweighting on stripes, dead bands and clouds. It should roughly tie on impulse noise. On clean data it should match unweighted STEREO within noise. A second check compares the low-rank weight tensor with unconstrained per-entry weights on impulse noise, where the constraint should prevent weights from chasing isolated outliers. A third check sweeps registration offsets from zero to a few pixels to locate the breakdown point, since misregistration mimics corruption in $\mathcal{R}$. A fourth check trains the unrolled variant on three corruption families and tests on the fourth. It should keep most of the optimization-based method's gain at a small fraction of its runtime. If it collapses on the unseen family, the fast path is not ready, and the optimization-based method remains the contribution.
References
- Hyperspectral Super-Resolution: A Coupled Tensor Factorization Approach
- Personalized Coupled Tensor Decomposition for Multimodal Data Fusion: Uniqueness and Algorithms
- Learning Tensor Low-Rank Prior for Hyperspectral Image Reconstruction
- Provable Dynamic Fusion for Low-Quality Multimodal Data
- EGM-Det: Entropy-Guided Multimodal Adaptive Fusion for UAV RGB-IR Object Detection
- RGB-T object tracking: Benchmark and baseline
- Interpretable Tensor Fusion