Buying Real Blur Pairs Where the Kernel Matters

Deblurring networks are trained mostly on synthetic blur. Real paired data, a blurry photo plus a pixel-aligned sharp photo of the same scene, has to be captured with special rigs, so each pair is expensive. Given a fixed budget of such pairs, which real scenes should be paid for? This note argues that the right choice follows from a known weakness of deblurring. Small errors in the estimated blur kernel get amplified into ringing and ghosting when the blur is undone. The blur kernel (or point spread function) is the small image that describes how the camera smeared each point. We propose to rank unlabeled real blurry images by how much their restored output changes when that estimated kernel is nudged in realistic ways. We then pay for ground truth only on images that are highly sensitive to the kernel yet not dominated by sensor noise or blown-out highlights.

The underlying bet is that the sim-to-real gap in deblurring has two kinds of causes. One kind is learnable: real camera-shake kernels and real kernel-estimation failures that synthetic training never showed the model. The other is largely irreducible: shot noise, clipping, and JPEG damage. Standard uncertainty-based active learning mixes the two and, as work in other domains shows, tends to spend its budget on the irreducible part. Kernel-perturbation sensitivity gives a physically motivated way to pull the learnable part out. If it works, a deblurring model adapted to real cameras should reach full-data quality with a fraction of the paired captures.

Core Claim

For pool-based selection of real paired blur data, we will rank images with a score that compares two quantities. The first is output sensitivity to realistic kernel perturbations. The second is output sensitivity to matched noise perturbations. We expect this score to beat both random selection and generic predictive-uncertainty selection in the number of real pairs needed to reach a target PSNR on real-blur benchmarks. We also expect the selected images to help even a restoration network that never estimates a kernel.

The idea comes from the frequency-domain view of inversion. With a blurry image $Y$, an estimated kernel $\hat K$, and the Fourier transform $\mathcal F$, a first-order expansion of naive inverse filtering shows two effects. A kernel error $\Delta K$ perturbs the restored spectrum by roughly $\mathcal F(Y)\,\mathcal F(\Delta K)/\mathcal F(\hat K)^2$, while noise $N$ perturbs it by $\mathcal F(N)/\mathcal F(\hat K)$. Both blow up where the kernel suppresses frequencies, as analyzed in Discriminative Non-blind Deblurring and Deep Learning for Handling Kernel/model Uncertainty in Image Deconvolution. However, only the kernel term scales with the image's own content at those frequencies. Taking the ratio therefore cancels the shared $1/\mathcal F(\hat K)$ amplification and leaves a quantity driven by scene content against noise level. Concretely, for a restoration operator $f(Y,\hat K)$ we define

$$ S(Y) = \frac{\mathbb E_{\Delta K \sim p_{\text{err}}}\,\lVert f(Y,\hat K+\Delta K) - f(Y,\hat K)\rVert^2}{\mathbb E_{\delta \sim \mathcal N(0,\sigma^2_{\delta})}\,\lVert f(Y+\delta,\hat K) - f(Y,\hat K)\rVert^2 + \epsilon}, $$

with the norms taken over pixels that are not saturated. Here $p_{\text{err}}$ is a distribution of realistic kernel-estimation errors: over-smoothing, missing kernel segments, and spurious spikes. These follow the erroneous-kernel synthesis of Nan and Ji. The noise scale $\sigma_\delta$ is matched to the perturbation energy of the kernel draws, and $\epsilon$ is a small constant. High $S$ marks textured content whose restoration hinges on getting the real kernel right. Images whose instability comes from noise get a low $S$ and are not selected.

Relation to Prior Work

Three groups of work motivate the proposal without doing it. The first makes deblurring robust to kernel error by training on synthetic perturbations. Nan and Ji unroll an optimization into network stages that predict the error caused by a wrong kernel, and train them on heuristic bad kernels (Deep Learning for Handling Kernel/model Uncertainty in Image Deconvolution). Schmidt et al. train cascades on random-walk trajectories mixed with perturbed kernels (Discriminative Non-blind Deblurring). Dong et al. embed a learned-feature Wiener filter that estimates its own noise ratio (Deep Wiener Deconvolution). All three use kernel uncertainty passively inside the model, and none decides what real data to collect. We reuse Nan and Ji's error model as $p_{\text{err}}$ and Dong et al.'s architecture as the scoring and restoration network.

The second group adapts to real degradation without labels. USR-DU learns a mean-and-variance model of real super-resolution degradation and samples from it passively (Learning Degradation Uncertainty for Unsupervised Real-world Image Super-resolution). The reblurring approach adapts each test image so that its output is hard to blur again (Clean Images are Hard to Reblur). That work reports perceptual gains but flat or falling PSNR, which is one reason we think some real labels are worth buying.

The third group supplies the active-learning lessons, and two of its results constrain our design. Bayesian active learning closed a sim-to-real detection gap with 36% of the real data (Bayesian Active Learning for Sim-to-Real Robotic Perception). The same paper needed randomized sampling to avoid label-distribution shift, and it degraded on the heavily imbalanced YCBV dataset, so we include a randomized diversity component and test for collapse. Peng et al. found that selecting by median rather than maximum aleatoric uncertainty avoids unlearnable outliers (Privacy-preserving and Uncertainty-aware Federated Trajectory Prediction). We treat that median-band rule as a competing way to dodge noise, not as our method. The file contains no prior work that uses kernel sensitivity as an acquisition signal. The closest work, the Bayesian sim-to-real paper, selects real images for detection using generic predictive variance.

Research Program

We will build the pipeline from three parts. A blind kernel estimator produces $\hat K$ for each pooled real image. A physics-informed restorer in the Deep Wiener style is first trained on synthetic GoPro-style blur and serves as both the scoring network and the first adapted model. A selection loop scores the unlabeled pool, queries labels in rounds, and fine-tunes.

Each round mixes two kinds of picks. Most picks are the top-$S$ images after excluding those whose saturated-pixel fraction exceeds a threshold. A minority are random draws, following the Bayesian sim-to-real finding. Computing $S$ costs about $M$ kernel draws plus $M$ noise draws per image. We will keep $M$ equal to the number of passes used by the MC-dropout baseline so that compute is matched.

The analysis has three parts. First, we will check whether $S$ actually separates the two causes of uncertainty. To do this, we correlate $S$ with each image's measured PSNR gain from fine-tuning on that image alone or in small groups. We compare this against the same correlation for predictive variance, and stratify by saturation fraction. RealBlur contains many low-light scenes with clipped lights, which is exactly where we expect maximum-uncertainty selection to go wrong. Second, we will train an end-to-end network, MIMO-UNet (Rethinking Coarse-to-Fine Approach), on the selected subsets. This tests whether the selection reflects the data rather than the scorer, which answers the objection that modern deblurring needs no kernel at all. Third, we will vary $p_{\text{err}}$ to see how much the method depends on a good kernel-error model.

Baselines

The naive reference is the synthetic-only model applied to real images with no real labels. Its counterpart is the same model fine-tuned on the entire labeled pool. Together these bound the curve, and "fraction of pool needed to recover the full-pool PSNR" is our main label-efficiency number.

Random selection at each budget is the baseline active learning most often fails to beat. It gets identical fine-tuning schedules and seeds, averaged over at least five runs. Beating it means a PSNR gain at small budgets that is larger than the spread across seeds.

The strongest generic competitor is predictive-variance selection. It uses MC-dropout or a small ensemble over the restorer's outputs, as in the Bayesian sim-to-real work, with the same randomized component and the same number of forward passes as our score. We will also run the median-uncertainty variant from Peng et al., since it is the existing remedy for noise trapping. Matching or exceeding both, especially on the low-light subset, is the core result.

A label-free baseline uses the reblur-consistency residual as the acquisition score. It reblurs the output with $\hat K$ and measures the mismatch with the input, which tests whether a simpler kernel-mismatch proxy is enough. We will also run zero-label adaptation: reblurring test-time adaptation from Nah et al., and synthetic kernel-error augmentation from Nan and Ji. These show whether paying for any labels is justified.

For absolute context on RealBlur-J, the strongest reported number in the file is EFE-CNA Net at 28.77 dB PSNR and 0.902 SSIM (EFE-CNA Net). The file does not state its training regime, so we treat it as a reference ceiling, not a matched comparison. MIMO-UNet and MPRNet were evaluated on RealBlur, but the file gives no RealBlur numbers for them.

Three ablations isolate the key ingredient: the kernel-sensitivity numerator alone without noise normalization, the noise-sensitivity term alone, and the full score without the saturation mask or the random component.

Why not just…?

Why not just randomize kernels and noise more widely in simulation? Domain randomization has a real track record. Still, the Bayesian sim-to-real paper used randomized BlenderProc data and still needed real active acquisition. Schmidt et al.'s randomized trajectories still lost ground on real noise. Our zero-label augmentation baseline tests this directly. If heavy randomization alone approaches the full-pool PSNR, the case for buying labels collapses, and we would report that.

Why not just use MC-dropout or ensemble uncertainty, which already works for sim-to-real? Predictive variance in deconvolution is inflated by the same $1/\mathcal F(\hat K)$ factor for noise as for kernel error. We therefore expect it to favor clipped, noisy night scenes, the trap Peng et al. describe. The saturation-stratified analysis will show whether this happens. If variance selection matches ours overall, the kernel-specific score adds nothing beyond interpretability.

Why not just use the reblur residual, which needs no perturbation passes? This is the most serious threat, because it is cheaper and also tracks kernel mismatch. Its weakness in principle is that noise and clipping also create reblur residuals. We run it as a full baseline at matched budget. If it performs as well as $S$, we will recommend it and present the sensitivity score as an explanation of why it works rather than a replacement.

Why not just adapt at test time and skip labels altogether? Reblurring adaptation runs iteratively on each image and reportedly leaves PSNR flat or lower while improving perceptual metrics. The comparison at zero labels against our method at a few percent of the pool will show how much distortion quality real labels buy.

Proposed Experiments

The main experiment simulates pool-based selection on RealBlur-J. We hide the labels of the training split, query at budgets of 2, 5, 10 and 25 percent of the pool, and evaluate PSNR and SSIM on the held-out test split. All runs start from a model trained only on GoPro, and RealBlur-R serves as a replication. Secondary checks on the Köhler et al. real camera-shake set and the Lai et al. set test transfer of the adapted models without further labels.

The decisive outcomes are four. First, the score must reach full-pool PSNR with substantially fewer labels than random selection and variance-based selection. A result in the range of the 36% reported for detection would be convincing, and parity with random would falsify the claim. Second, its advantage over variance selection should be concentrated on high-saturation and low-light test images. Third, subsets chosen by $S$ should also improve the kernel-free MIMO-UNet more than random subsets of the same size. Fourth, the ablation dropping the noise denominator should lose most of the gain. If that last check fails, the "separate learnable from irreducible" story is wrong, even if the selection happens to work.