Phone cameras build sharper photos by shooting a rapid burst of frames. Tiny involuntary hand shake shifts each frame by a fraction of a pixel, so together the frames sample the scene more finely than any single one does. This only works where the scene holds still. When something in the frame moves, such as a passing cyclist, a swinging sign, or a falling ball, current burst methods treat it as an alignment error and fall back to one frame there. We propose to stop discarding moving objects and to use a basic piece of physical common sense instead. Over the roughly tenth of a second a burst lasts, a moving object does not jump around at random. It follows a smooth path set by inertia, close to constant velocity or constant acceleration. If motion is modeled as a few trajectory parameters per point, shared across all frames, rather than as an independent motion field per frame pair, then every frame contributes evidence to the same numbers. That pooled estimate should be precise enough at the sub-pixel level to fuse moving objects the way static backgrounds are fused today.
The model has three parts. The first is a motion model that splits burst motion into an unconstrained per-frame camera shake term and a dense, low-order trajectory field for scene motion. The second is a single weight-shared update block, applied repeatedly, that refines those trajectory coefficients. The third is a reconstruction network trained end to end with the motion module. The bet is that the physical prior does two jobs. It fixes the known failure of end-to-end-trained motion, which drifts into unphysical "task-oriented" flow. It also supplies the noise-averaging that sub-pixel estimation on noisy low-resolution frames lacks.
Core Claim
Burst and video super-resolution pipelines estimate motion frame pair by frame pair. Each pairwise estimate must find a sub-pixel shift from only two aliased, noisy observations. That is the "resolution conflict": the fine detail needed to measure a sub-pixel shift is the same detail missing from the low-resolution input. We claim that, for moving objects within a burst, writing motion as a trajectory turns $N-1$ separate ill-posed problems into one well-posed fit. Concretely, the displacement of scene point $x$ at frame time $t_i$ relative to the reference frame is
$$ w_i(x) = g_i(x) + v(x)\,t_i + \tfrac{1}{2}a(x)\,t_i^2 + r_i(x), $$
where $g_i$ is a low-dimensional per-frame camera motion (hand tremor, left free because it is effectively random), $v$ and $a$ are dense velocity and acceleration fields shared across all frames, and $r_i$ is a small residual whose size is penalized. Under a constant-velocity model with independent per-frame localization noise of standard deviation $\sigma$, the least-squares velocity estimate has error that falls roughly as $\sigma/(\sqrt{N}\,T)$, where $T$ is the burst duration; this follows from ordinary linear regression over $N$ time samples. A pairwise estimator gets no such benefit. The new claim is that this pooling, learned end to end under a reconstruction loss and refined by a shared iterative operator, recovers real detail on moving objects. Existing methods mask these regions out, so they can only hallucinate or blur them.
Relation to Prior Work
Two lines of work motivate the gap. Handheld Multi-Frame Super-Resolution showed that hand tremor alone supplies enough sub-pixel shifts for joint demosaicking and super-resolution. Deep Burst Super-Resolution built a learned RAW burst pipeline on a frozen, pre-trained PWC-Net flow estimator, with an attention module that down-weights misaligned pixels. In both, independently moving objects count as misalignment to be suppressed, and we target exactly that discarded region. On the joint-optimization side, Video Enhancement with Task-Oriented Flow. trained a SpyNet flow module end to end with restoration networks. It found that the resulting flow beats geometrically accurate flow but becomes distorted and unphysical, and that it fails to transfer across tasks. This finding constrains our design: we keep end-to-end training but restrict the motion module's freedom to a physically motivated form, and we test whether this preserves its gains while keeping velocity faithful.
The method draws on three other papers. Learning for Video Super-Resolution through HR Optical Flow Estimation (SOF-VSR) estimates flow directly on the high-resolution grid with a sub-pixel convolution output. We adopt its high-resolution motion output, but its flow is pairwise over three frames, while ours is a single trajectory over the whole burst. Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation showed that one shared-weight block applied repeatedly beats stacked distinct networks, with 26.4% fewer parameters and about 18% better Sintel accuracy than PWC-Net. It is trained for two-frame geometric error only and never jointly with a downstream task. Our update block follows its residual-refinement form but updates trajectory coefficients and is trained through reconstruction. On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding found that conditioning a shared block on its iteration index lifts a weight-tying bottleneck, so we include iteration-index conditioning. Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks showed that shared-weight networks can run more iterations at test time to solve harder inputs, but also that running too many can degrade results. We test both effects on longer bursts and faster motion.
The physical common sense literature motivates the prior. Physion: Evaluating Physical Prediction from Vision in Humans and Machines found that models reading physics from raw pixels lag far behind models given true physical state. Interpretable Intuitive Physics Model decoded a physics bottleneck into optical flow to predict frames. Both work at coarse, synthetic, pixel scale and ignore sensor sampling. We go the other direction. We use only the most elementary physical regularities, inertia and spatiotemporal continuity, but apply them at the sensor's sub-pixel scale on real RAW data. We make no claim about collisions, support, or the higher-level reasoning those benchmarks probe.
Research Program
We will build a RAW burst model with four parts. An encoder packs Bayer frames as in Deep Burst SR. A motion state holds per-frame camera parameters $g_i$, dense high-resolution fields $v$ and $a$, residuals $r_i$, and a per-point confidence map. One shared update block reads features warped by the current trajectory and outputs increments to all of these, for $K$ iterations conditioned on iteration index. A fusion and decoder stage then combines the aligned frames, weighting them by confidence. Training uses only reconstruction loss plus a magnitude penalty on $r_i$. There is no flow supervision.
Real motion sometimes breaks the low-order model, for example at collisions, with non-rigid deformation, or at occlusions. The confidence map and residual let the model fall back to attention-style masking there rather than forcing a wrong trajectory. We will measure how often this fallback fires, because if it fires everywhere the prior is doing nothing.
Training data will be synthetic bursts made from real video with real object motion. We take Vimeo-90K clips, add random sub-pixel camera tremor, downsample, mosaic, and add calibrated shot and read noise. Evaluation of velocity fidelity uses bursts synthesized the same way from MPI Sintel, which has ground-truth flow. Real-data evaluation uses BurstSR.
The analysis has three parts. First, we measure how velocity error at the high-resolution grid scales with burst length $N$ and noise level, and compare the curve to the $1/\sqrt{N}$ regression rate and to the flat curve expected from a pairwise estimator. Second, we measure whether learned velocities stay physically faithful, by end-point error against Sintel ground truth, in contrast to the drift reported for task-oriented flow. Third, we measure how reconstruction quality and velocity error change as the number of test-time iterations $K$ varies beyond its training value.
Baselines
The naive reference point is single-frame super-resolution on the reference frame. On BurstSR, Deep Burst Super-Resolution reports 44.02 dB PSNR for its single-image baseline. Inside dynamic regions this is what masking-based burst methods effectively fall back to, so it is the floor our moving-object results must clear by a clear margin.
The strongest existing burst method in the file is Deep Burst SR itself, with frozen PWC-Net alignment and attention fusion; it reports 45.17 dB PSNR on BurstSR. It is the right main competitor because its attention fusion is exactly the mechanism that suppresses moving objects. We will retrain it on our synthetic dynamic bursts with the same data, the same RAW encoder and decoder, a matched parameter and compute budget, and the same hyperparameter search budget. Beating it means a statistically reliable PSNR gain inside masks of independently moving pixels on synthetic dynamic bursts, with no loss on BurstSR overall (that is, not below 45.17 dB under its protocol). Because BurstSR scenes are largely static, we do not expect large overall gains there, and we will not claim them.
The strongest joint-optimization competitor is a TOFlow-style model, as in Video Enhancement with Task-Oriented Flow.. It has the same architecture, but the motion module outputs free per-frame flow fields and is trained end to end. For video comparisons we also run SOF-VSR, which reports 26.01 dB on Vid4 at 4x. Beating these means matching or exceeding their reconstruction PSNR while achieving lower end-point error against true velocity.
Four ablations isolate the key ingredient, all at matched compute. The first keeps everything but replaces the trajectory model with free per-frame flow refined by the same shared block. This is an IRR-style operator trained jointly, and it isolates the trajectory parameterization. The second keeps free per-frame flow but adds a soft penalty on temporal second differences, tuned over a grid. The third unrolls $K$ unshared blocks instead of one shared block. The fourth removes iteration-index conditioning.
Why not just…?
Why not just train a shared-weight iterative flow network end to end, without any physical prior? This is the most direct alternative: IRR-style refinement plus TOFlow-style training. Prior evidence says end-to-end flow without constraints drifts into unphysical vectors that serve as reconstruction shortcuts and do not transfer Video Enhancement with Task-Oriented Flow.. That evidence concerns physical fidelity, not PSNR, so it does not settle the question. The first ablation answers it directly. If it matches our dynamic-region PSNR and our velocity error-versus-$N$ curve, the trajectory prior adds nothing and we will say so.
Why not just add a temporal smoothness penalty to ordinary multi-frame flow? This is a real threat. With a large, well-tuned weight, a soft penalty on second differences approaches the hard constant-acceleration model. Our case for the hard form has two parts. It shrinks the unknowns from $N-1$ dense fields to two, so each iteration of the shared block accumulates evidence from all frames into the same quantities. It also needs no weight to be tuned per noise level. Neither part is guaranteed to matter in practice. The second ablation, given an equal tuning budget, decides it. If the soft penalty ties, the contribution narrows to the camera-versus-scene split and the iterative solver, and we will rescope accordingly.
Why not just use an existing multi-frame flow estimator? VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation already shows that combining several frames resolves ambiguous two-frame matches. However, it is trained for geometric accuracy on synthetic data and is not built for aliased, noisy RAW input at high-resolution output. We will plug it in as a frozen aligner in the Deep Burst SR pipeline. If it closes most of the dynamic-region gap, the benefit comes from using more frames rather than from the physical form or from joint training.
Why not just keep attention masking, since moving regions are small? For typical handheld photos, dynamic pixels may be a small fraction of the image. In that case global PSNR barely moves, and masking is a perfectly good engineering answer. This is why our primary metric is PSNR restricted to moving-object regions and text and texture legibility on moving objects. We also report the fraction of image area affected, so readers can judge the practical stakes.
Proposed Experiments
The decisive experiment is a controlled sweep on synthetic dynamic RAW bursts. We vary burst length (4 to 14 frames), noise level, and object speed and acceleration. For each setting we measure dynamic-region PSNR and high-resolution velocity end-point error for our model, Deep Burst SR, the TOFlow-style model, and the four ablations. The prediction that would confirm the idea has two parts. Our velocity error should fall with $N$ at close to the regression rate while pairwise baselines stay flat. Dynamic-region PSNR should rise with $N$ for our model and stay near the single-frame floor for masking-based fusion. A flat curve for our model refutes the central claim.
The second experiment tests physical fidelity under end-to-end training on Sintel-derived bursts. The question is whether reconstruction-only training keeps learned velocities close to ground truth for our model while the TOFlow-style model drifts.
The third experiment probes the prior's limits. We use bursts containing collisions or abrupt direction changes, and we measure how often the confidence fallback fires and whether output quality degrades gracefully to the masking baseline rather than below it.
The fourth experiment varies test-time iterations $K$ above and below the training value. It checks for the gains from extra computation and the overthinking failure reported by Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks, and whether iteration-index conditioning stabilizes the result.
Finally, we report BurstSR and Vid4 under their standard protocols. These check that the method costs nothing on mostly static scenes and are not evidence for the core claim.
References
- Handheld multi-frame super-resolution
- Deep Burst Super-Resolution
- Video Enhancement with Task-Oriented Flow.
- Learning for Video Super-Resolution Through HR Optical Flow Estimation
- Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation
- On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding
- Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks
- Physion: Evaluating Physical Prediction from Vision in Humans and Machines
- Interpretable Intuitive Physics Model
- VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation