Held-Out Burst Frames as a Ground-Truth-Free Test of Restored Detail

When you restore a photo, standard scores like peak signal-to-noise ratio (PSNR, a log-scaled pixel error against a clean reference) punish sharp, correct detail that lands a fraction of a pixel away from the reference. They also reward blur. Learned "naturalness" scores have the opposite flaw: they reward invented texture. Neither can tell whether fine detail in a restored image is real or made up. A handheld burst, meaning several quick shots of the same scene taken while the hand shakes slightly, already contains a better judge. Each frame samples the scene at a slightly different sub-pixel offset. Fine detail the sensor cannot resolve therefore shows up in each frame as a distinct aliasing pattern, the false coarse pattern that too-fine detail folds into, and that pattern depends on the frame's exact offset. Suppose we hide some frames from the restoration method. We can then render its output through a simple camera model into each hidden frame's viewpoint and check whether the predicted aliasing matches what the sensor actually recorded. Real detail predicts the hidden frames. Invented detail with the wrong spatial phase does not.

We propose this held-out-frame consistency score as an evaluation method, not a new restoration network. It needs no ground truth, is insensitive to global sub-pixel misregistration, and applies across burst tasks: super-resolution, denoising, and extreme low-light enhancement. It addresses the problem that clouds comparisons between regression and generative restorers, namely that the field cannot measure faithfulness separately from either pixel distortion or plausibility. It also gives a principled way to choose how much generative sharpening a burst model is allowed to apply.

Core Claim

Our central claim is that likelihood of held-out raw frames, computed under a low-capacity camera model, ranks restoration methods by faithfulness of high-frequency detail. It does this on real bursts where clean references are missing or misaligned, and in a way that neither PSNR nor perceptual metrics do. The claim has a built-in falsification condition. The score's power to separate real from invented detail should come specifically from sub-pixel motion. It should therefore collapse on bursts with zero relative motion, which is the "stabilization null" noted by Handheld Multi-Frame Super-Resolution. It should grow with the amount of aliased energy in the frames. If the score keeps discriminating without motion, it is detecting something other than measurement consistency, and the claim fails.

Concretely, let $\hat{x}$ be a restored high-resolution linear image produced from input frames $I$, and let $U$ be the held-out raw frames $b_j$. We define

$$C(\hat{x}) = \frac{1}{|U|}\sum_{j \in U} -\log p\big(b_j \mid M\,S\,K\,W_{\phi_j}\hat{x}\big),$$

where $W_{\phi_j}$ warps $\hat{x}$ by a motion $\phi_j$ estimated once per burst (shared by all methods being compared), $K$ is a small lens blur, $S$ is the downsampling, $M$ is the Bayer mosaic, and $p$ is a calibrated Poisson–Gaussian sensor noise model. Before scoring, only a global two-parameter translation and the few blur parameters are refit per output. This capacity cap is deliberate. A flexible learned forward model could absorb errors in $\hat{x}$ and hide hallucinations. We report $C$ relative to the noise floor, which is the expected value of $C$ if $\hat{x}$ were the true scene. We also report a frequency-split version that isolates residuals in the bands where aliased high-resolution content lands.

How This Relates to Prior Work

The mechanism comes directly from Self-Supervised Burst Super-Resolution. That work splits a burst into input and unseen frames and trains a super-resolution network by rendering its prediction through a learned warp–blur–downsample–mosaic model. It reaches 38.81 dB on SynBSR versus 36.84 dB for weakly paired supervision. We reuse the held-out-frame idea but change its role from training loss to an independent evaluation applied to other methods' outputs. That change forces design decisions their work never faced. Motion must be fixed per burst rather than per method. Forward-model capacity must be capped so the score cannot be gamed. The score must be extended beyond super-resolution to noise and low-light gain. Their own noted limitation, a spatially invariant blur with rigid shifts, constrains us too. We treat it as a measurable source of score bias rather than assume it away.

Three papers motivate why such a score is needed. Deep Burst Super-Resolution aligns burst features with optical flow and fuses them. It reports 45.17 dB versus 44.02 dB for single-image super-resolution on BurstSR and a 53.6% best-ranked user preference. It also had to train with a contextual loss, because slight phone–DSLR misalignment made pixel losses penalize true detail. That is direct evidence that the reference itself is unreliable on real data. Efficient Burst Super-Resolution with One-step Diffusion adds a generative stage on top of Burstormer, gaining texture at the cost of PSNR. It openly carries the risk of invented texture when motion is ambiguous, and it offers no way to measure that risk. Is Nano Banana Pro a Low-Level Vision All-Rounder? documents the split in which a generalist generator wins human and no-reference scores but trails specialists by over 4 dB PSNR on DIV2K super-resolution and 5–8 dB on denoising. It attributes this partly to metric sensitivity to small shifts and partly to real hallucination (identity swaps, corrupted text). Current metrics cannot separate those two explanations, and our score is built to do so.

Two papers frame the metric side and act as our comparison points. Perceptual Fidelity Aware Mean Squared Error (PAMSE) blurs both images before squared error to mimic human contrast sensitivity, reaching 0.9162 Spearman rank correlation on TID2008 versus 0.7718 for MSE. BiRQA builds a fast, attack-robust full-reference quality model, with Spearman 0.988 on LIVE and 0.979 on CSIQ. Both still need a clean, aligned reference. Both are built to predict human opinion rather than physical correctness. PAMSE in particular discards exactly the high frequencies whose faithfulness we want to judge. The Perception-Distortion Tradeoff supplies the theory. Its tradeoff concerns the gap between expected distance to a ground truth and the realism of the output distribution. Extra measurements change the posterior and thus where the frontier lies. Our score gives an operational way to see how far a burst method can move toward realism while staying consistent with its measurements.

The cross-task regime is set by Burst Image Restoration and Enhancement (BIPNet) and Burstormer. Both apply one burst architecture to super-resolution, denoising and low-light enhancement, with separate weights per task, and judge everything by PSNR. Single-image all-in-one models such as PromptIR and FPro serve here as competitors that cannot use sub-pixel evidence. We deliberately do not build the obvious all-in-one burst network. We think the more basic bottleneck is that nobody can measure whether such a network's detail is real, so we address measurement first.

Research Program

The work has three parts. The first is building the scorer. This includes per-burst motion estimation on the high-resolution grid using all frames, following the reasoning of Learning for Video Super-Resolution through HR Optical Flow Estimation that low-resolution flow is too coarse; there, high-resolution-grid flow cut endpoint error from 1.11 to 0.45. It also includes per-sensor noise calibration and the capped forward model. A task-specific term is added only where the physics demands it: an exposure gain and black level for low-light, and nothing extra for denoising.

The second part is validation under controlled conditions, where the true answer is known. We tamper with outputs in two opposite ways. Some edits should hurt faithfulness: replacing the top frequency band with phase-randomized or diffusion-sampled texture of matched energy. Others should not: whole-image sub-pixel shifts, or mild global blur matched in PSNR loss. The score must penalize the first kind and largely ignore shifts, a pattern PSNR, LPIPS, PAMSE and no-reference metrics each fail in a predictable direction.

The third part is using the scorer. We re-rank existing burst, single-image all-in-one, and generalist restorers on real data across three tasks. We then use the score to pick the operating point of a generative burst model.

Baselines

The naive reference point is the evaluation the field uses now: PSNR and SSIM against the aligned reference under the standard BurstSR and SyntheticBurst protocols, with no held-out frames. On the restoration side, the strongest reported numbers in our file are Burstormer's 42.83 dB PSNR on SyntheticBurst (Burstormer) and DBSR's 45.17 dB on real BurstSR (Deep Burst Super-Resolution). For E-BSRD we found no reported LPIPS or FID value on Lacuna. These restorers are the subjects being scored, not competitors to the score. Every restorer, including single-image PromptIR applied to the reference frame and the generalist model from LowLevelEval, receives exactly the same input frames, with the same frames withheld. No method is retuned.

The strongest competing metrics are BiRQA (Spearman 0.988 on LIVE, BiRQA) and PAMSE (Spearman 0.9162 on TID2008, Perceptual Fidelity Aware Mean Squared Error). We also include a shift-searched PSNR that takes the best value over a small grid of sub-pixel translations, plus LPIPS. These are the strongest competitors because they are what a careful researcher would use today to argue a sharp output is faithful. We run all of them with the same reference images and the same alignment. No-reference scores (NIQE, MUSIQ, NIMA) are included as the plausibility-only reference.

The score beats these baselines if it meets three conditions. First, in the tampering test it separates hallucination edits from shift and blur edits with clearly higher area under the curve. Second, on synthetic data its method ranking agrees with a ground-truth faithfulness target at least as well as the reference-based metrics, without using the reference. Third, on real data it agrees better than they do with a content-correctness human study.

Two ablations isolate the key ingredient. The first replaces held-out frames with the model's own input frames, since a method can trivially fit what it has already seen. The second compares only at low resolution, without modeling phase-dependent aliasing, which removes the sub-pixel mechanism. A third ablation swaps our capped forward model for a flexible learned one, as in Self-Supervised Burst Super-Resolution, to show whether flexibility hides hallucinations.

Why not just…?

Why not just use shift-tolerant PSNR or LPIPS against the reference? On synthetic data with exact references this is a real threat. A shift-searched reference metric may detect invented detail about as well as our score. Our synthetic experiment is designed to show this and we will report it plainly. The value of our score is claimed only where the reference is missing, misaligned, or itself a single random realization: real BurstSR, where Deep Burst Super-Resolution had to abandon pixel losses because of misalignment, low-light captures, and in-the-wild bursts with no reference at all. If shift-searched PSNR matches our score's agreement with the human content-correctness study on real BurstSR, the claimed advantage shrinks to "works without a reference."

Why not just use a robust learned or no-reference quality metric such as BiRQA or MUSIQ? These predict human opinion scores. The LowLevelEval study shows such scores favor a generator that swaps identities and corrupts text. Plausible invented texture is exactly what they are trained to like. BiRQA's robustness to adversarial attack is a different property from detecting confident but wrong detail. The tampering test checks this directly: we expect no-reference metrics to score diffusion-sampled texture better than the truth.

Why not just train with the held-out-frame loss, as Bhat et al. did, and skip the metric? Training with the loss does not tell you whether a different method, or a generative add-on, is faithful. A score used for both training and evaluation becomes a target that methods are tuned toward and so stops measuring anything independent. We keep the evaluator separate and low-capacity. We test whether the evaluator can then guide training only in a downstream experiment, choosing E-BSRD's noise level, that is reported separately.

Why not just check consistency with the input frames? A method can match its inputs exactly while inventing detail between their sampling positions. The input-frame ablation should show weak separation in the tampering test. If it does not, held-out frames add nothing and we would report that.

Proposed Experiments

The validation stage uses SyntheticBurst, with its known references and synthetic shifts, and the burst denoising data used by BIPNet. Each burst is split into input frames and two to four held-out frames. We apply the tampering edits to outputs of DBSR, BIPNet, Burstormer and E-BSRD, and measure separation of faithfulness-breaking edits from shift and blur edits for every metric. The faithfulness target is band-limited, shift-compensated error against the true reference. We will also regenerate bursts with motion scaled toward zero and with increasingly strong optical blur. The decisive check is that our score's separating power should fall toward chance with no motion and shrink as blur suppresses aliasing. If it does not, the score is measuring something other than measurement consistency.

The real-data stage uses BurstSR (14-frame Samsung bursts with misaligned DSLR references) and the SID Sony low-light subset. Methods are re-ranked by every metric. We run a human study that asks raters to verify content rather than preference: reading text and counting fine repeated structures in crops, checked against the DSLR capture. We report Kendall rank agreement between each metric and these correctness judgments. We predict our score will rank E-BSRD's texture above Burstormer's only where the texture predicts held-out frames. We also predict it will place per-frame PromptIR and the generalist model below burst methods by more than PSNR alone suggests.

The final experiment sweeps E-BSRD's added-noise level and a perceptual-loss weight on Burstormer. We ask whether the point of greatest sharpness before $C$ rises above its noise floor matches the human-verified boundary at which detail stops being correct. If it does, the score gives burst restoration a usable stopping rule for generative sharpening that requires no ground truth.