Measurement-Aware Zero-Cost Scoring for Real-Time Snapshot Spectral Decoders

A snapshot spectral camera freezes motion by recording every wavelength in a single exposure. It passes light through a patterned mask and a prism, so that all bands land on the sensor at once, overlapped. The price is paid in computation: a neural network has to untangle that overlapped image back into a full spectral cube. It must also do this fast enough to keep up with the moving scene the camera was built for. The best decoders are hand-designed and stack many repeated stages. Nobody has a cheap way to find smaller decoders that are nearly as good. "Zero-cost proxies," which score an untrained network from one forward and backward pass, work well for hyperspectral classification, where they even succeed on pure noise inputs. There is a reason to doubt they transfer to reconstruction. In reconstruction, a network's quality depends on the specific measurement process it is inverting, and a proxy that ignores the mask and prism is scoring the wrong thing.

We propose a zero-cost score conditioned on the measurement process, together with the benchmark needed to test whether it matters. The score has two parts. It feeds the untrained decoder real simulated measurements. It then computes its gradient statistics only on the part of the reconstruction error that the measurements cannot constrain, meaning the component of the spectral cube invisible to the camera. The hypothesis is that decoders differ mainly in how they fill in that invisible component. Everything the measurement already pins down is handled by the data-consistency steps every such decoder shares. If the hypothesis holds, the score should rank decoders better than size-biased generic proxies at matched compute. That would allow a search, taking minutes, for decoders that meet a video-rate latency budget.

Background and the gap

In coded aperture snapshot spectral imaging (CASSI), a cube $F$ is masked, sheared by a prism, and summed onto a 2D sensor, giving $Y=\Phi F+e$ with $\Phi$ far wider than tall. A standard property of this operator is that $\Phi\Phi^\top$ is diagonal. Projecting any cube onto the null space of $\Phi$, the set of cubes the camera cannot see, therefore costs one elementwise division: $P_N = I-\Phi^\top(\Phi\Phi^\top)^{-1}\Phi$.

The reconstruction literature supplies the model family. TSA-Net introduced separate spatial and spectral self-attention. It also showed that training without simulated shot noise causes collapse on real hardware, which is evidence that the measurement statistics matter. DAUHST unrolls an iterative solver into stages. Each stage performs a data-consistency step with degradation parameters estimated from $Y$ and the mask, then applies a transformer denoiser that splits channels between local-window and shuffled attention. It reports 38.36 dB PSNR on KAIST with 9 stages. It also reports that its 2-stage variant beats the previous best model, MST-L, with 31% fewer parameters and 34% fewer FLOPs. All of these configurations are hand-set. TMCA learns time-varying masks jointly with a fixed U-Net decoder and reports about 1 second per frame. That is far from video rate.

Two papers constrain what we build. STAS is the closest prior work on search. It trains a one-shot supernet over eight convolutional operators for hyperspectral restoration, including CASSI on KAIST. It finds a 1.43M-parameter model that beats TSA-Net's 44.3M. However, it has no attention, no unfolding stages, a fixed mask, and still requires supernet training. HyTAS benchmarks 12 zero-cost proxies on 2,000 hyperspectral vision transformers. It proposes ZiCo++ and reports Spearman correlations of 0.73 on Indian Pines and 0.75 on Houston2013. It finds that proxies rank well on Gaussian-noise inputs, and it documents a bias toward attention modules. Its task is classification of already-captured cubes.

NAS-Bench-Suite-Zero supplies the generic proxy suite. It finds nwot and flops the most robust individual proxies, and it documents the general bias of proxies toward bigger models. That size bias is exactly what would make a proxy recommend 9 stages when a latency budget allows 2. Our proposal differs from HyTAS by conditioning on the operator $\Phi$ and targeting inversion. It differs from STAS by being training-free and covering attention and unfolded search spaces.

Core claim

Architecture quality for snapshot spectral decoders is operator-dependent, and a proxy that isolates null-space error recovers most of the ranking signal. Concretely, for an untrained decoder $f_\theta$ with mask $M$, we sample a few CAVE cubes, simulate $Y=\Phi F+e$ with Poisson noise, and compute

$$S(f)=\sum_{\ell}\frac{1}{|\theta_\ell|}\log\frac{\mathbb{E}\,|\nabla_{\theta_\ell}\mathcal{L}_N|}{\mathrm{std}\,(\nabla_{\theta_\ell}\mathcal{L}_N)},\qquad \mathcal{L}_N=\|P_N(f_\theta(Y)-F)\|^2 .$$

Here $\theta_\ell$ is the weights of layer $\ell$, and the ratio is the ZiCo-style gradient signal-to-noise. Dividing by $|\theta_\ell|$ normalizes each module by its own size. This targets both HyTAS's attention-dominance problem and the general size bias. Removing the range-space error discards the part that unrolled data-consistency steps fix regardless of architecture. We expect that part to be what inflates scores for deeper stacks.

Research program

We first build the benchmark. None exists for reconstruction, and without one every claim is untestable. The search space generalizes DAUHST. It varies the number of stages from 1 to 9, per-stage width, the local-versus-shuffled channel split, window size, heads, and feed-forward ratio. It also allows each stage's denoiser to be replaced by STAS-style separable convolutions, so the convolutional and attention families share one space.

We sample about 300 architectures and train each on CAVE, testing on the 10 KAIST scenes, under the standard fixed mask. A subset of 60 is also trained under a second random binary mask and under a learned mask. The learned-mask condition runs on the cheaper TMCA setting: ICVL, 256×256 crops, 12 bands.

Full DAUHST schedules are expensive. Ground truth therefore uses a shortened schedule. We verify on 20 architectures that shortened-schedule rankings agree with full-schedule rankings, and report the resulting Kendall τ. For every architecture we measure latency on one edge GPU at KAIST resolution. The motivating constraint is 33 ms per frame, the video-rate regime where snapshot capture actually pays off.

We then score every architecture with our proxy, its ablations, and the baselines, and run a latency-constrained search. The search uses evolutionary sampling over the full space and is scored by the proxy. It is followed by full training of the top five candidates.

Baselines

The naive reference point is to pick the largest hand-designed DAUHST variant that fits the latency budget, typically 2 or 3 stages, with no search. This is what a practitioner would do. DAUHST reports that its 2-stage model beats MST-L at lower cost. The file gives no exact 2-stage PSNR, so this reference is retrained in our pipeline. The accuracy ceiling is DAUHST-9stg at 38.36 dB PSNR on KAIST, as reported in that paper. We rerun it on our splits so the numbers are comparable.

The strongest ranking competitor on paper is ZiCo++ from HyTAS (ρ = 0.73 and 0.75 on classification). We run it exactly as specified, on noise inputs and on real measurements.

Alongside it we run the 13 generic proxies from NAS-Bench-Suite-Zero, notably flops and nwot. We also run their XGBoost combination, which that paper reports improves Spearman correlation by up to 42%. For the combination, all surrogate methods get the same 20 to 50 trained labels.

The strongest real competitor is budgeted short training following Rethinking Performance Estimation in Neural Architecture Search: each candidate is trained for $k$ epochs, and we sweep $k$ so its GPU-hours match or exceed ours. We also run STAS-style one-shot search over the same space, with a supernet budget equal to the total short-training budget.

Three ablations isolate the key ingredient. The first uses the full loss instead of the null-space loss. The second uses noise inputs instead of simulated measurements. The third scores with a mismatched mask.

Beating the baselines means two things. First, higher Spearman correlation than every zero-cost baseline within FLOP-matched bins, which removes size as an explanation. Second, a searched decoder under 33 ms that exceeds the naive budget-matched DAUHST variant by at least 0.5 dB on KAIST. The 0.5 dB threshold is a margin we set ourselves, not a reported figure.

Why not just…?

Why not just pick the stage count by FLOPs? In DAUHST, stage count clearly dominates accuracy, and it may dominate everything. We test this directly by measuring PSNR spread among architectures within the same FLOP and latency bin. If that spread is below about 0.3 dB, search inside a bin is pointless. We would then report that result and reduce the contribution to the benchmark plus the latency frontier.

Why not just train each candidate briefly? This is the most serious threat, because short training sees the real operator and real data for free. Our claim is only that the proxy matches short training's ranking at roughly 1/100 of its compute. That makes the proxy useful as a prefilter over thousands of candidates. The sweep over $k$ will show where short training overtakes us. If it wins at every budget we can afford, the proxy fails.

Why not reuse HyTAS's ZiCo++ as is? HyTAS observed data independence on classification. Reconstruction quality, by contrast, hinges on measurement noise and the mask, as TSA-Net and DAUHST show. The noise-input and mismatched-mask ablations test exactly this. If ZiCo++ on noise ranks as well as our score, operator conditioning is unnecessary and we will say so.

Why not a STAS supernet extended with attention? Weight sharing across paths with very different depths and attention layouts is known to corrupt rankings, and supernet training on 3D cubes is costly. We run it at matched budget rather than assume it loses.

Decisive checks

The whole program rests on three measurements. The first is the Kendall τ between architecture rankings under two different masks. If τ is near 1, rankings do not depend on the operator. The motivation for conditioning on $\Phi$ then collapses, and the proxy reduces to a debiased generic score. The second is whether the null-space loss beats the full loss within FLOP bins. This is the single test of the core hypothesis. The third is whether the searched sub-33 ms decoder beats the naive DAUHST variant at equal latency, on simulation and on the real CASSI scenes used by DAUHST. If the first two results come out positive and the third fails, the result is still useful: operator-aware scoring then works for ranking but does not yet change what gets deployed.