Context Only Where the Camera Is Blind: Multi-Scale Spatial Context for the Null Space of RGB-to-Spectrum Reconstruction

An ordinary camera records three numbers per pixel, but the light hitting it has a full spectrum. Spectral reconstruction tries to recover that spectrum, typically 31 bands from 400 to 700 nm, from a single RGB image. The task is ill-posed because of metamerism: many different spectra produce the same RGB triplet. Neural networks resolve the ambiguity partly from spatial context. A pixel's neighbourhood hints at what material it is and under what light.

Context has a cost, though. Pooling information over a wide area can blur the boundary between two materials that look nearly the same but differ spectrally. The best current models, MST++ and SSHFormer, apply spatial context to the entire output spectrum. That includes the part the camera already measured exactly.

This proposal splits the spectrum into two parts. The first is the component the camera actually sees, which can be computed per pixel from RGB in closed form. The second is the null-space component the camera is blind to, and only this part is predicted with multi-scale spatial context. The context comes from a block that mixes several dilation rates, meaning convolution kernels with gaps that see wider neighbourhoods at no extra parameter cost. A per-pixel gate chooses how much context each location uses.

The bet has two parts. Restricting context to the blind component should prevent boundary smearing in the visible component. The learned scale gate should reveal how much spatial context metamer resolution actually needs.

Core Claim

Physically Plausible Spectral Reconstruction from RGB Images decomposes a spectrum $r$ as $r = r_{\parallel} + r_{\perp}$. Here $r_{\parallel} = S^{+} c$ is fixed by the observed RGB $c$ and the camera sensitivity matrix $S$, with $S^{+}$ its pseudo-inverse, and $r_{\perp} = B a$ lies in the null space of $S$, spanned by basis $B$ with coefficients $a$. That paper trains a simple per-image convolutional network to predict only $a$, which guarantees zero colour error. It never asks how spatial context should enter that prediction.

Our claim is that this decomposition is the natural place to resolve the open tension between SegFormer and TransDeepLab. SegFormer argues that a hierarchical transformer makes parallel multi-scale context modules redundant. TransDeepLab rebuilds such a module, a spatial pyramid of attention windows, and finds it useful. For an inverse problem, we predict both are right, but for different parts of the signal. The visible component needs no context at all. The blind component needs scale-adaptive context.

Concretely, we predict three things. A network that applies multi-rate dilated context only to $a$ will match or beat a spatial-spectral hybrid on reconstruction error. It will reduce errors concentrated at material boundaries. Its scale gate will prefer large context in exactly the regions where the RGB value is least informative: dark, saturated, or low-texture pixels.

Research Program

The backbone reuses the efficient spectral-token attention of MST++. That attention treats each feature channel as a token, so its cost grows linearly with image size, and it sits inside MST++'s U-shaped encoder–decoder hierarchy. We change two things.

The first change is in the feed-forward sublayer. MST++ uses a plain 3×3 convolution there, and SSHFormer uses a single fixed dilation. We replace it with a channel-partitioned multi-rate block. Following PSConv, channels are split into groups with dilations of 1, 2, 4 and 8, so the parameter count stays the same as a plain convolution. Interleaving dense and sparse rates, as in Augmented ASPP, avoids the checkerboard sampling gaps that a single large dilation leaves. A lightweight per-pixel softmax gate then weights the scale groups before fusion. This is a cheap form of the attention-filtered aggregation that Zhou et al. found necessary to keep diffuse context from swamping faint local signals.

The second change is the output head. It emits only null-space coefficients $a$, and the final spectrum is the analytic $S^{+} c$ plus $B a$. Because $S^{+} c$ scales exactly with exposure, we also feed the network intensity-normalised RGB and rescale $a$ by pixel brightness. This makes the prediction equivariant to exposure by construction, rather than through the log-exposure augmentation Lin and Finlayson needed.

The hard part is real-world data. ARAD-1K inputs pass through a non-linear camera pipeline and JPEG compression, so the clean linear camera model $S$ does not hold. The NTIRE 2022 report documents roughly doubled error in this regime. We handle it with a learned linearisation: a small per-channel tone-curve network followed by a global 3×31 matrix $\hat S$, both fit to the training pairs. This is close in spirit to the sensitivity estimation of Kaya et al..

On ARAD-1K, the zero-colour-error guarantee therefore holds only with respect to $\hat S$. The experiments measure how much this approximation costs. On CAVE and ICVL with simulated linear RGB, the guarantee is exact.

Baselines

The naive reference point is the analytic visible component alone, $S^{+} c$, which has no learning. A second naive point is a per-pixel MLP that predicts $a$ from the RGB value with no spatial context. Together they show how much error is metameric and how much spatial context recovers.

The strongest efficient competitor is MST++. On ARAD-1K it reports 34.32 dB PSNR, MRAE (mean relative absolute error) of 0.1131 and RMSE of 0.02308 (MST++). It is the closest architecture to ours, differing only in spatial mixing and output parameterisation.

SSHFormer reports 34.39 dB on the same benchmark (SSHFormer). It is the direct test of whether our multi-rate gated block beats a single-dilation hybrid. That paper quotes the MST++ baseline as 34.10 or 34.32 dB in different places, so we will retrain both rather than trust cross-paper numbers.

Diff-Spectra reports the highest ARAD-1K figure in the file, 35.47 dB PSNR, but no MRAE or RMSE. It relies on iterative test-time diffusion and extra synthetic metameric data. We will report it as an upper reference rather than a like-for-like competitor, and compare against it on PSNR only.

On CAVE, FRN reports 41.05 dB with 0.30M parameters. We will adopt its 20/6/6 split. On ICVL, HSCNN-Rpd reports ΔE = 0.00 and a half-exposure MRAE of 2.83, against 15.68 for the unconstrained HSCNN-R (Physically Plausible SR).

Fairness requires that all retrained models share several things: the 900/50/50 ARAD split, the same patch-sampling and augmentation pipeline, a matched parameter budget within ±10% of MST++, and an equal number of hyperparameter-search trials. Latency is measured on the same GPU.

Beating the strongest baseline means one of two outcomes. The first is a mean ARAD-1K MRAE below retrained MST++ and SSHFormer across three seeds. The second is equal MRAE with a clear reduction in boundary-region error and exposure-shift error.

The key ablations form a 2×2 design. One factor is single-scale versus multi-rate gated spatial mixing. The other is full-spectrum versus null-space-only output. Together they separate what the scale mixture contributes from what the output restriction contributes.

Why not just…?

Why not just bolt an ASPP-style pyramid head onto MST++? This is the textbook multi-scale context fix, and SegFormer's evidence suggests a hierarchical encoder already makes such heads redundant for segmentation. We include MST++ with an ASPP-style head in the ablations. Our hypothesis concerns where context is applied, not whether a pyramid exists. If the ASPP head on a full-spectrum output matches our null-space model, including on boundary error, the "context only where the camera is blind" claim is falsified.

Why not just use larger kernels instead of dilation? HSRA mixes 3×3 and 9×9 depth-wise kernels for compressive spectral imaging and reports 37.49 dB on KAIST. It has not been tested on RGB input. Large kernels avoid the irregular memory access that PSConv reports, about 1.06× latency. This alternative is a real threat on hardware efficiency, so we run a matched variant with parallel 3×3 and 9×9 depth-wise kernels. If it matches accuracy, we will report the large-kernel version as the recommended design. The null-space and gating findings would stand regardless of which spatial operator is used.

Why not just project MST++'s output onto the RGB constraint after training? This is the most serious alternative. A post-hoc projection that replaces the visible part of any prediction with $S^{+} c$ also gives zero colour error, at no architectural cost. The question is whether training a network to predict only $a$ changes what it learns, for example by concentrating its spatial context on ambiguous regions, or whether it is equivalent to projecting afterwards. We test this directly by comparing post-hoc-projected MST++ and SSHFormer with our trained null-space model. If the gap is within seed noise, we will say the output restriction contributes only a guarantee and not accuracy. The contribution then narrows to the multi-rate gated block and the scale-need analysis.

Proposed Experiments

The main comparison runs on ARAD-1K, using validation for model selection and reporting MRAE, RMSE and PSNR. It also uses the NTIRE out-of-scope track of human subjects under studio light, to probe generalisation. On CAVE (20/6/6), Harvard (30/10/10) and ICVL (100/50/50), we simulate linear RGB with known sensitivities, so the decomposition is exact. On these we test cross-camera transfer by training with CIE 1964 colour-matching functions and testing with Sony IMX135 sensitivities, as in Lin and Finlayson.

Three checks would decide the question.

The first is a boundary analysis. We compute MRAE separately on pixels within 3 px of ground-truth spectral edges, defined as large spectral-angle gradients, and on interior pixels. The hypothesis predicts that the null-space model's advantage is concentrated at boundaries.

The second is a scale-gate analysis. We correlate the gate's mean dilation per pixel with the per-pixel metameric error of the context-free MLP baseline. A positive correlation supports "context resolves metamers". A flat gate supports SegFormer's view that a hierarchy alone suffices.

The third is an exposure sweep from 0.25× to 4× on ICVL and ARAD. It tests whether built-in exposure equivariance holds MRAE flat, where an unconstrained baseline would show the reported jump from 1.76 to 15.68.

We will also report the colour error ΔE with respect to the learned $\hat S$ and against true held-out RGB on ARAD. This makes explicit how much of the guarantee survives a non-linear real camera pipeline.