Keeping a Denoising Slider Calibrated While the Camera Drifts

Many image restoration systems now ship with a control knob. A user, or a downstream program, slides a value to choose between a smoother, cleaner image and a sharper one that keeps more texture and more noise. These knobs are calibrated once, at training time, against a fixed noise distribution. Real cameras do not hold still statistically. Sensors heat up, gain and ISO change, white balance and lighting shift, and a phone may hand a stream from one sensor to another. After such a change, the same knob position means something different. A setting that was balanced in the morning over-smooths at noon, and nobody notices, because no clean reference image ever arrives.

This proposal treats the meaning of the knob as the quantity that drifts, and keeps it calibrated from the noisy video stream alone. The method builds on a controllable denoiser whose output is a weighted sum of a few precomputed noise components. For that kind of model, a label-free estimate of restoration error built from pairs of frames turns out to be a quadratic function of the knob weights. It can be minimized in closed form on a CPU and updated incrementally frame by frame. An unsupervised drift detector decides when to discard old statistics. The user's knob is then re-anchored to the current best setting rather than to a fixed absolute value, so "a little smoother than ideal" stays true as the camera changes.

Core Claim

Real-Time Controllable Denoising for Image and Video (RCD) supplies the mechanism we reuse. A backbone such as NAFNet or Restormer predicts $L$ noise maps of different strengths. A decorrelation step makes these maps nearly orthogonal, and any denoising strength is then a cheap linear combination $\hat{x}_t(w) = y_t - \sum_{l=1}^{L} w_l n_{t,l}$. Here $y_t$ is the noisy frame at time $t$, $n_{t,l}$ is its $l$-th noise map, and $w \in \mathbb{R}^L$ is the control. RCD's weights come from a user or from a heuristic AutoTune that assumes the noise matches training.

Our observation is that a second noisy view $y'_t$ of the same content gives an unbiased error signal. This view can be an adjacent aligned video frame or another frame of a burst. Its noise must be zero-mean and independent of the noise in $y_t$; the Noise2Noise argument then gives $\mathbb{E}\|\hat{x}_t(w) - y'_t\|^2 = \mathbb{E}\|\hat{x}_t(w) - x_t\|^2 + \text{const}$. Because $\hat{x}_t$ is linear in $w$, the windowed risk is a quadratic with a closed-form minimizer:

$$ w^\star = G^{-1} b,\qquad G_{lk} = \sum_{t \in \mathcal{W}} \langle n_{t,l}, n_{t,k}\rangle,\qquad b_l = \sum_{t \in \mathcal{W}} \langle n_{t,l},\, y_t - y'_t\rangle, $$

where $\mathcal{W}$ is the current window of frames. $G$ is $L\times L$ and nearly diagonal because RCD decorrelates the maps. The sufficient statistics take $O(L^2)$ inner products per frame, so the solve is negligible next to the backbone's forward pass, which already runs. The user control becomes a relative offset, $w(u) = (1+u)\,w^\star$, where $u = 0$ is the current estimated best point.

What is new is not drift detection, and not self-supervised denoising. It is the claim that a controllable restoration model with a linear control space can be recalibrated under drift by an exact streaming least-squares problem, with no backpropagation. A second claim is that re-anchoring keeps the semantics of a fixed user setting stable across distribution changes.

Relation to Prior Work

The closest work adapts restoration at test time, but none of it maintains a calibrated control across a stream. Test-Time Degradation Adaptation for Open-Set Image Restoration (TAO) optimizes a small degradation adapter per image inside a 1000-step diffusion sampler. It resets for every image and exposes no control. We keep memory across frames and replace iterative optimization with a closed-form solve. Test-Time Preference Optimization for Image Restoration (TTPO) steers outputs with frozen no-reference quality models. An Underwater Image Enhancement Benchmark Dataset and Beyond shows that such metrics diverge from human judgment on hard images, which constrains our design. We therefore use a risk estimate whose bias we can analyze rather than a learned quality score.

Controllable Continual Test-Time Adaptation motivates the worry that unconstrained online updates accumulate error. It uses "control" to mean geometric regularization of classifier features, a different sense from ours. We sidestep representation collapse by never updating backbone weights; only $L$ coefficients move.

The control-space papers motivate the gap. CFSNet, Searching for Controllable Image Restoration Networks and RCD all assume a stationary mapping from knob to outcome. CFSNet and TASNet have nonlinear control paths, which is why we build on RCD's linear one. Extending the idea to them would require a local linearization. We treat that as a stretch goal, not a claim.

From the drift literature we take detectors. Fast Unsupervised Online Drift Detection Using Incremental Kolmogorov-Smirnov Test (IKS) gives $O(\log N)$ univariate Kolmogorov-Smirnov updates. Its univariate restriction is normally a weakness, but here it fits well: RCD's decorrelated components give $L$ nearly independent scalar streams, the per-frame normalized projections $b_{t,l}/G_{t,ll}$. Sequential Covariate Shift Detection Using Classifier Two-Sample Tests provides a sequential classifier test as an alternative detector, with a known cost: it needs on the order of 200 samples of warm-up at a 1% false-positive level.

Research Program

We first build the estimator and characterize its bias. The Noise2Noise identity fails when noise is temporally correlated, which processed camera pipelines can introduce through temporal denoising or compression. It also fails when alignment between $y_t$ and $y'_t$ is imperfect, since misalignment adds content error that pulls $w^\star$ toward stronger smoothing. We will derive the bias under a correlation coefficient $\rho$ between the two noise draws. It shrinks $b$ by a factor tied to $\rho$, so the method under-denoises predictably. We will measure this bias empirically and add a guard: optical-flow confidence masks restrict the inner products to well-aligned pixels.

Next comes the windowing policy, which is where drift handling enters. Under stationarity the estimate should use a long window to reduce variance. After a change, old statistics are harmful. We will compare three policies for discarding them: a tuned fixed exponential forgetting factor, IKS run on the $L$ projection streams with a Bonferroni-corrected threshold, and the sequential classifier test. On detection, the system resets $G$ and $b$ and warm-starts $w^\star$ from the previous value.

Then comes control re-anchoring. We test whether $u$ applied relative to $w^\star$ yields consistent excess error and consistent appearance across regimes, compared with RCD's absolute weights.

Baselines

The naive reference is RCD with its weights frozen at the value that is best on the training distribution, with no adaptation. This is what a deployed system does today.

The second baseline is RCD's own AutoTune, run exactly as released on the same backbone.

The strongest competitor is a learned per-frame weight predictor. This is a small network trained on clean/noisy pairs to regress the oracle $w$ from the noisy frame, with training data augmented over all noise levels and camera types available in training. It reacts instantly and needs no detector, so it is the method our approach must beat on held-out regimes. It gets the same backbone and the same training data, and its augmentation range is tuned on a validation split for the same budget we spend tuning our forgetting factor and detector threshold.

We also run an online Noise2Noise fine-tuning of the whole backbone on the stream. This is the heavy test-time-training route in the spirit of Test-Time Training with Masked Autoencoders, matched in frames seen. We report its compute separately.

Normalization-statistic adaptation in the style of Zero-Shot Anomaly Detection via Batch Normalization is included where the backbone has such layers.

An oracle that picks the best $w$ per regime using clean references gives the upper bound. The ablation that isolates the drift ingredient is our estimator with a tuned fixed forgetting factor and no detector.

For context, Restormer reports surpassing 40 dB PSNR on SIDD (Restormer). RCD reports matching its backbone's PSNR/SSIM with a 2000x reduction in editing latency, but the file gives no RCD table values. We found no result on Lacuna for any controllable denoiser on a drifting stream.

Beating the strongest baseline means two things on held-out cameras or noise shapes: average PSNR at least 0.3 dB closer to the per-regime oracle than the learned predictor, and no loss larger than 0.05 dB on in-distribution segments.

Why not just…?

Why not just predict the knob from each frame with a trained regressor? This is the real threat. If noise drift were only a change in level, a well-augmented regressor might track it perfectly. Our argument is that drift also changes noise shape: spatial correlation from demosaicking, ISP-dependent coloring, and clipping effects that A High-Quality Denoising Dataset for Smartphone Cameras documents as camera- and condition-specific. Under such changes the regressor's learned map from appearance to best weight is itself stale, which is concept drift in the strict sense. The held-out-camera experiment settles this. If the regressor stays within 0.1 dB of the oracle there, the drift machinery is unnecessary, and we will report that outcome as such.

Why not just use a fixed forgetting factor and skip drift detection? This is the classical adaptive-filtering answer, and it may be nearly as good. Any fixed factor trades lag after abrupt changes against variance during stable stretches. A detector can choose long windows in stable periods and short ones after a change. The ablation measures this directly as recovery time in frames and as steady-state PSNR variance. If detection does not improve both, we drop it.

Why not just fine-tune the denoiser online with a self-supervised loss? Per-frame backpropagation costs orders of magnitude more than a $L\times L$ solve, and the cost of test-time training is acknowledged in the TTT-MAE work. Continual updates also risk the error accumulation documented by C-CoTTA. We will show the quality–compute frontier of the two approaches; online fine-tuning may win on quality for large shape changes.

Why not just let the user move the slider? Users cannot see slow drift. At video rates there is no verification signal in time, which is the verification-latency problem IKS was designed around. Our re-anchoring keeps the user in control rather than removing them.

Proposed Experiments

Drifting streams come from two sources. The first is SIDD. Its five phones, three brightness levels and three color temperatures provide multiple real noisy frames per scene, which serve as independent second views. We concatenate segments with abrupt camera and ISO switches and with gradual interpolation of illumination, and we hold out two phones entirely for the regressor-generalization test. The second source is the video benchmarks RCD used, DAVIS and Set-8. On these we synthesize noise whose level and spatial correlation change over time, abruptly and gradually. We also run a variant with controlled temporal correlation $\rho$ to probe the estimator's failure mode.

Metrics are PSNR and SSIM against the per-regime oracle, recovery time after each change, detector delay and false-alarm rate, CPU latency per frame, and control consistency. Control consistency is the variance across regimes of the excess error at a fixed $u$, compared with the same variance at a fixed absolute RCD weight.

Four checks decide the outcome. First, under stationarity the method must match the frozen RCD weights, so it must not hurt. Second, on held-out cameras it must beat the learned per-frame predictor by the margin stated above. Third, detector-gated windowing must beat the best fixed forgetting factor on recovery time without worse steady-state variance. Fourth, the measured bias under correlated noise must match the derived prediction, which tells practitioners when the method is safe to use.