Modern optical flow networks estimate how every pixel moves between two frames by starting from a guess and repeatedly correcting it. At each step a small recurrent network looks up how well the current guess matches the second image, then proposes a correction. This design, introduced by RAFT, is usually credited with self-correction: a bad early guess gets repaired by later steps. The literature on cascaded image pipelines points the other way. When an early stage leaves artifacts, later stages treat them as real signal and amplify them. A refinement loop is also a cascade, only with shared weights. Under real sensor noise, such as photon shot noise in dark raw images or fog returns in LiDAR, every match lookup the loop consumes is corrupted. So the loop may drift instead of converge.
This note proposes to settle that question directly, and to fix the failure if it exists. We measure flow error at every refinement step on real low-light raw data. We then give the recurrent updater a per-pixel estimate of how unreliable each match lookup is. That estimate is computed from a calibrated physical noise model of the sensor, not learned from scratch, and it lets the network shrink its corrections where the evidence is noise. The claim can be tested and can fail. If per-step error curves stay monotone under heavy noise, the loop self-corrects, and physics-based training augmentation alone is the right answer.
Core Claim
Our hypothesis is that recurrent flow refinement shows a signature of step-wise error compounding under structured sensor noise. Error falls for the first few steps and then rises again, in regions where the match signal is dominated by noise. We expect this to persist even after the network is trained on physically realistic noise. If so, it can be reduced by conditioning each update on the sensor's analytic noise variance.
The noise model comes from Optical Flow in the Dark. In that model a raw pixel with clean intensity $x$ is observed as
$$y \sim \mathcal{N}\left(x,\ \sigma_s^2 x + \sigma_r^2\right),$$
where $\sigma_s^2$ scales photon shot noise and $\sigma_r^2$ is read noise from the amplifier and analog-to-digital converter. Both are calibrated per camera and gain setting. From the observed $y$ we form a per-pixel variance map $v = \sigma_s^2 y + \sigma_r^2$ for each frame. We then compute a reliability map $r_t$ that combines the variance at the source pixel with the variance at the location the current flow points to. The recurrent unit receives $r_t$ alongside its usual inputs and predicts a per-pixel step gate $g_t \in [0,1]$:
$$f_{t+1} = f_t + g_t \odot \Delta f_t.$$
Here $f_t$ is the flow after step $t$, $\Delta f_t$ is RAFT's proposed correction, and $\odot$ is per-pixel multiplication.
The new element is not noise-aware training; that exists. It is feeding a physics-derived, per-lookup uncertainty into every iteration of the learned optimizer. This targets the specific way an iterative method can compound error.
Relation to Prior Work
Three papers motivate the question. RAFT builds an all-pairs match volume and runs a weight-shared recurrent updater over a single high-resolution flow field. That design removed the coarse-to-fine failures of image pyramids. It was trained and evaluated on clean or well-lit data. Deep Bayesian Video Frame Interpolation likewise argues that iterative inference progressively repairs corrupted initial motion. On the other side, Deep Residual Network for Joint Demosaicing and Super-Resolution, Gated Fusion Network and Crafting a Toolchain for Image Restoration show that serial stages amplify upstream artifacts. None of these papers looks inside an iterative flow solver. We turn their disagreement into a per-step measurement.
Two papers are the closest existing work and constrain the design. Optical Flow in the Dark shows that denoise-then-estimate pipelines break motion consistency, and that training directly on physics-synthesized raw noise helps. It uses PWC-Net, a pyramid method, and uses the noise model only to make training data. Multi-Modal Synergistic Implicit Image Enhancement enhances features implicitly before a RAFT-style solver. It trains with simple synthetic noise rather than a calibrated sensor model, and its authors note that performance varies across cameras. Both treat the solver as a fixed black box that receives better inputs. We keep inputs raw and change what the solver knows at each step.
Adverse Weather Optical Flow adapts flow to rain and fog by aligning match-volume statistics between synthetic and real weather. It does not examine how the updater behaves across iterations. We take its point that match-volume statistics are the right place to act, and act per step instead of per domain.
Two papers supply methods for the 3D extension: FOGGIFY, a physical fog simulator for LiDAR, and PV-RAFT, a 32-iteration recurrent scene flow method for point clouds. QS³ and AdaptPoint supply the evidence that physically structured corruption beats Gaussian jitter. Both are static classification studies with no motion or iteration.
Research Program
The first stage is diagnostic. We train RAFT in two ways. One version uses clean FlyingChairs and FlyingThings data. The other uses physics-noise training data built with the recipe that produced FlyingChairs-DarkNoise (FCDN). We evaluate both on the Various Brightness Optical Flow (VBOF) benchmark at all exposure levels, unrolling up to 32 iterations.
For each exposure we record end-point error (EPE, the average pixel distance between predicted and true flow) after every step. We also record the norm of each update and the drift of the hidden state. These curves are split by a per-pixel noise-to-signal ratio computed from $v$. The compounding hypothesis predicts a specific pattern. Error curves should be U-shaped in high-variance regions and monotone in low-variance ones. The turnaround step should move earlier as exposure drops. We will report whichever pattern appears.
The second stage builds the calibrated updater described above. We compute $r_t$ by bilinearly sampling the two variance maps at the source pixel and the flow endpoint, downsampling to RAFT's 1/8 resolution, and passing the result through a small convolution. The output joins the match features entering the recurrent unit. The gate head adds a few thousand parameters.
Training uses the same physics-noise data as the strongest baseline. Noise parameters are drawn over the calibrated range of the three VBOF camera brands. At test time we use each camera's calibrated parameters. We also hold out one brand entirely to test calibration transfer.
The third stage transfers the idea to 3D, which is where the literature has no data. We apply FOGGIFY to the real LiDAR point clouds in KITTI scene flow across a sweep of fog densities. Ground-truth flow is kept for surviving hard-target points, and ghost returns get no ground truth. We run PV-RAFT with and without a per-point reliability input. The reliability comes from FOGGIFY's range-dependent attenuation and backscatter probability, with the fog coefficient either given or estimated from return-intensity statistics.
This stage has a limitation. No real foggy scene flow benchmark with ground truth appears in the file, so the 3D test is simulation applied to real clouds. It is not real fog.
Baselines
The naive reference is RAFT trained on clean synthetic data and applied to raw VBOF frames. It shows how badly an unmodified recurrent solver fails. Next to it we run the cascade RAFT would face in practice: an enhancement network, as in the See-in-the-Dark comparison of Optical Flow in the Dark, followed by clean RAFT. It is included because it is what engineers deploy.
The strongest competitor is RAFT trained on physics-synthesized raw noise. This is the Optical Flow in the Dark recipe moved onto an iterative backbone. It uses the same noise model and training data as our method and differs only in lacking the reliability input and gate. It gets the same schedule, iteration count, GPU budget and hyperparameter search. On PWC-Net, Optical Flow in the Dark reports a VBOF Exp-6 EPE of 6.09 with physics-noise training, against 10.88 without. No reported result was found on Lacuna for RAFT on VBOF, so we produce that number ourselves.
We also run the implicit feature enhancement of Multi-Modal Synergistic Implicit Image Enhancement. It reports EPE 3.85 against RAFT's 4.41 on noised FlyingThings3D. We compare on that setting and on VBOF, using their released model if available and a matched reimplementation otherwise.
Beating the strongest baseline means a consistent EPE reduction at the lowest three VBOF exposures, across all three camera brands and three training seeds. It also means a flatter per-step error curve in high-variance regions, with no loss on well-exposed frames.
The ablations isolate the physics ingredient. The first gives the gate a learned confidence map from match features alone, with no variance map, so the network must infer reliability itself. The second applies a fixed damping schedule that shrinks steps uniformly. The third feeds deliberately miscalibrated noise parameters from the wrong camera. If the miscalibrated version does as well as the calibrated one, the gain is not coming from physics.
In 3D, the baselines are PV-RAFT trained clean and PV-RAFT trained with FOGGIFY augmentation, both under identical budgets. The clean reference on real clear-weather KITTI is PV-RAFT's reported EPE of 0.0560 m with a 21.63% outlier rate. No reported result was found on Lacuna for scene flow under simulated or real fog.
Why not just…?
Why not just train RAFT on physics-based noise and stop there? This is the real threat. It may well capture most of the gain, since it roughly halved PWC-Net's error on VBOF Exp-6 in Optical Flow in the Dark. That is why it is the primary baseline and not a strawman. The diagnostic stage decides the matter. If the noise-trained RAFT already shows monotone per-step error in high-variance regions, compounding is absent, and we will report that augmentation suffices and the gate is unnecessary. The gate is justified only if a turnaround survives noise-aware training and the gate removes it.
Why not just denoise first and then run RAFT? Two direct comparisons found the cascade harmful for flow: Optical Flow in the Dark and Multi-Modal Synergistic Implicit Image Enhancement. The restoration literature (Gated Fusion Network, Joint Demosaicing and Super-Resolution) explains why: downstream stages treat upstream artifacts as detail. We still run the cascade on VBOF with RAFT, because it has not been measured with an iterative backbone.
Why not let the network learn its own confidence, with no physics? A learned gate might find reliability from match-volume shape alone. The learned-confidence ablation tests exactly this. The physics map should win mainly on the held-out camera and at extreme exposures, where match-volume statistics are least informative. If the two tie everywhere, we drop the physics claim and keep the simpler gate.
Why not just run fewer iterations? If error turns around, stopping early is the cheapest fix. The per-step curves give the best fixed stopping point for each exposure, which we report as a baseline. Our prediction is that the best stopping point differs across pixels within a single image. A spatially varying gate can exploit that and a global iteration count cannot. The claim is falsified if the best fixed early stop matches the gated model.
Decisive Checks
Four results decide the project. The first is whether noise-trained RAFT shows a per-step error turnaround on VBOF Exp-5 and Exp-6, concentrated in pixels with high noise-to-signal ratio. If it does not, the proposal reduces to a negative result about compounding, which is still useful for resolving the disagreement in the literature. The second is whether the calibrated gate beats both noise-trained RAFT and the learned-confidence gate on the held-out camera brand. The third is whether the miscalibrated physics map erases the gain. The fourth is whether the same ordering holds for PV-RAFT on FOGGIFY-corrupted KITTI clouds, where backscatter ghosts create systematically false matches rather than random ones.
If all four hold, the result goes beyond "use realistic noise." It says that iterative solvers need to be told, at every step, how much to trust what they see. It also gives a cheap, physically grounded way to tell them.
References
- Optical Flow in the Dark
- RAFT: Recurrent All-Pairs Field Transforms for Optical Flow
- Deep Bayesian Video Frame Interpolation
- Deep Residual Network for Joint Demosaicing and Super-Resolution
- Gated Fusion Network for Joint Image Deblurring and Super-Resolution
- Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning
- Multi-Modal Synergistic Implicit Image Enhancement for Efficient Optical Flow Estimation
- Adverse Weather Optical Flow: Cumulative Homogeneous-Heterogeneous Adaptation
- Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather
- PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clouds
- Quasi-Balanced Self-Training on Noise-Aware Synthesis of Object Point Clouds for Closing Domain Gap
- Sample-adaptive Augmentation for Point Cloud Recognition Against Real-world Corruptions