Many computational cameras do not record an image directly. They record a scrambled, compressed measurement and rely on a reconstruction algorithm that knows exactly how the hardware scrambled it. A coded aperture snapshot spectral imager (CASSI) is one example: a patterned mask and a prism squash a 28-band spectral cube into a single 2D photo. The algorithm's description of the hardware is a linear "sensing matrix." Networks trained in simulation assume this matrix is exact, but in a real instrument the mask sits slightly off position, the prism disperses light nonlinearly across wavelengths, and pixel gains drift. These are not additive noise. The error scales with the signal itself, which produces an error floor that more training noise cannot remove (General Deviants, Robustly Stable Signal Recovery with Structured Matrix Perturbation). For these instruments, a large share of the sim-to-real gap is plausibly a gap in the sensing matrix, not in image appearance.
We propose a reconstruction network that treats this gap as a small set of physical unknowns. The unknowns are a few mask-shift, dispersion-curve and gain parameters. The network estimates them from the measurements while reconstructing. Because these parameters are the same for every photo the instrument takes, the network also pools its estimates across all captures. The idea comes from blind calibration theory: a single snapshot cannot pin down the hardware error, but a handful of snapshots that share it can (Cal-AMP). We also propose a ground-truth-free way to score real-hardware reconstructions with numbers, which the field currently lacks.
Core Claim
Two claims follow. First, an unrolled reconstruction network that explicitly identifies a low-dimensional perturbation $\Delta$ will transfer to unseen hardware errors better than two alternatives: a network that learns an unconstrained residual operator, and a network that has only been shown randomized perturbations during training. Unrolling means turning the steps of an iterative solver into network layers. The structured model is $A = \hat{A} + B\Delta$, where $\hat{A}$ is the nominal CASSI matrix, $B$ maps physical parameters to matrix changes, and $\Delta$ holds roughly tens of numbers.
Second, pooling $\Delta$ across $P$ captures from the same instrument should produce a sharp improvement between $P=1$ and small $P$. This mirrors the phase transition proven for Cal-AMP, which is impossible at $P=1$ and near-oracle by $P\approx 10$ under analytic priors. One part of this is testable rather than assumed: whether a learned image prior already makes some structured perturbations identifiable from one snapshot. Mask shifts are a candidate, because the mask's high-frequency pattern should make a shift visible in a single image. Per-pixel gains are not a candidate, because a scene's brightness and a pixel's gain trade off in a way a single image cannot separate.
Relation to Prior Work
Three deep CASSI methods motivate the work and set the bar. All three report real-hardware results only qualitatively.
- RDLUF-MixS² replaces the ideal matrix in each gradient step with a learned module for the "residual degradation." That module is an unconstrained network shared across all inputs. It is not tied to physical parameters and is not estimated per instrument.
- DAUHST predicts step sizes for each iteration from the mask and the measurement. It keeps the matrix itself fixed.
- PSR-SCI guides a diffusion model with an operator it assumes is exact.
We differ from all three by making the perturbation an explicit, identifiable, instrument-level quantity.
Two lines of work supply the method:
- The structured-perturbation model and its alternating estimation scheme come from AA-P-BPDN (Robustly Stable Signal Recovery) and from structured sparse total least-squares (S-TLS). Total least-squares treats the matrix itself as uncertain, not just the measurements.
- The expectation-maximization update for the calibration parameters follows BAd-VAMP. BAd-VAMP nearly closes the calibration gap in tomography, reaching 38.27 dB against a perfectly calibrated 39.57 dB, but it uses analytic or BM3D denoisers.
We unroll this alternating scheme with a learned spectral denoiser.
Knowledge distillation under inexact operators is the closest work framed as sim-to-real. It trains a student network to work with a perturbed matrix by imitating a teacher trained on the exact one. It never estimates the perturbation and is tested only on MNIST and synthetic MIMO detection.
NEAR avoids pretraining by optimizing per snapshot. We keep amortized inference, meaning a single forward pass per image.
Research Program
The work has three parts.
Perturbation family. We define a physically grounded family $B\Delta$ for SD-CASSI. It contains a sub-pixel mask translation and rotation, a low-order polynomial correction to the per-band dispersion shift, a smooth per-column gain field, and a small blur kernel for the optics. Training uses KAIST/CAVE scenes with $\Delta$ drawn from a prior.
Network. We build an unrolled network in the style of DAUHST, with each stage alternating between two updates:
- a learned-denoiser image update under the current matrix $\hat{A} + B\hat{\Delta}$;
- a $\Delta$ update.
The $\Delta$ update is a Gauss–Newton step on the measurement residual $\|y_p - (\hat{A}+B\Delta)x_p\|^2$. Here $y_p$ and $x_p$ are the measurement and current image estimate for capture $p$, and the residual is summed over the $P$ captures in a pool. A small learned network corrects this step.
Identifiability analysis. For each component of $\Delta$, we ask whether the Fisher information from one capture is full rank or degenerate, given the learned prior. This predicts which parameters need pooling. We then check the prediction empirically by sweeping $P$.
Real-data scoring. We introduce a held-out-pixel test. The method reconstructs from a random 90% of detector pixels and must predict the remaining 10% through its own estimated operator. Error on the held-out pixels is a quantitative real-data score that needs no hyperspectral ground truth. The holdout matters because a method that merely overfits a flexible residual operator to the measurement would score well on pixels it saw. We validate this score in simulation by checking that it ranks methods the same way true PSNR does.
Baselines
Naive reference. DAUHST-9stg trained on the nominal operator and applied unchanged. It reports 38.36 dB PSNR on the 10 KAIST scenes (DAUHST). Its loss under perturbation measures the size of the gap.
Strongest existing method. RDLUF-MixS², reported at 39.57 dB on the same unperturbed scenes (RDLUF-MixS²). It is the only method that already learns a matrix correction, so it is the direct competitor. No reported result was found on Lacuna for any method on KAIST under perturbed operators. We will retrain RDLUF on exactly our perturbation-randomized data, with the same number of stages and a comparable parameter count and training budget. Beating it means higher PSNR and SSIM on held-out perturbation magnitudes and families, plus lower held-out-pixel error on real SD-CASSI captures. Matching it in-distribution and winning out-of-distribution would still support the claim.
Classical self-calibration. BAd-VAMP with a BM3D denoiser on the same pooled captures. This tests whether the learned prior is necessary.
Calibration oracle. Our own network given the true $\Delta$. This sets the ceiling the estimation step should approach.
Ablations isolating the key ingredients.
- Explicit structure. Our exact architecture and randomized training data, with the $\Delta$-update removed. This isolates whether explicit identification matters or randomization alone suffices.
- Pooling. The full model run at $P=1$.
Distillation baseline. We also run the distillation method on its own Single-Pixel Camera (SPC) MNIST benchmark. There it reports up to 1.5 dB PSNR gain at σ=0.3 over a non-distilled student (KD). Its perturbations are unstructured Gaussian, so we expect our structural advantage to vanish there. That makes it a negative control: it shows the gain comes from correct structure, not from extra capacity.
Why not just…?
Why not just calibrate once in the lab and plug in the measured matrix? This is a real threat for a stable instrument. The calibration oracle shows how much it would buy if calibration were perfect. The open question is how fast the true $\Delta$ drifts after lab calibration. Cal-AMP's motivation explicitly cites drift and in-field changes, and RDLUF reports a visible gap remaining even with a measured mask. To settle it, we will run lab-calibrated DAUHST and our method on real captures taken across sessions and compare held-out-pixel error. If lab calibration matches our method across sessions, the pooling claim fails for CASSI. We would then reframe the method as calibration-free setup.
Why not just train with heavy randomization of perturbations and skip estimation? This is the most likely threat. The "explicit structure" ablation tests it directly with identical data and compute. Theory predicts randomization alone fails at large perturbations: the error grows with $\|x\|$, so a network that does not know $\Delta$ must average over many possible operators, which leaves a floor (General Deviants). Our prediction is that the gap between the ablation and the full model grows with perturbation magnitude. If it does not, we concede the point.
Why not just optimize $\Delta$ at test time through a frozen pretrained network? This is simpler and close in spirit to NEAR's per-snapshot fitting. We include it as a baseline: gradient descent on pooled measurement residuals through the oracle-trained network. If it matches the unrolled estimator at acceptable runtime, the contribution narrows to the structure-plus-pooling claim and the scoring test. That would still be useful, and we would report it as such.
Decisive Checks
There are four decisive checks.
- On KAIST with perturbation families held out from training, the full model should close most of the gap between the naive reference and the oracle, and should beat both retrained RDLUF and the no-estimation ablation.
- The $P$-sweep should show a sharp rise for gain-type parameters but not for mask shift. This would confirm the identifiability analysis.
- In simulation, the held-out-pixel score must rank methods the same way true PSNR does. Otherwise it is not used on real data.
- On real SD-CASSI captures, the method should lower held-out-pixel error relative to DAUHST and RDLUF. This would give the first quantitative real-hardware comparison in this line of work.
References
- General Deviants: An Analysis of Perturbations in Compressed Sensing
- Robustly Stable Signal Recovery in Compressed Sensing With Structured Matrix Perturbation
- Blind Calibration in Compressed Sensing using Message Passing Algorithms
- Residual Degradation Learning Unfolding Framework with Mixing Priors Across Spectral and Spatial for Compressive Spectral Imaging
- Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
- Spectral Compressive Imaging via Unmixing-driven Subspace Diffusion Refinement
- Sparsity-Cognizant Total Least-Squares for Perturbed Compressive Sampling
- Bilinear Recovery using Adaptive Vector-AMP.
- Learning to Reconstruct Signals With Inexact Sensing Operator via Knowledge Distillation
- NEAR: Neural Electromagnetic Array Response