Long-range cameras for border, maritime, and wildlife surveillance look through kilometers of heated air. Atmospheric turbulence, meaning random bending of light by temperature fluctuations, makes every pixel jitter by a few pixels from frame to frame. The targets these systems care about are often only 3–15 pixels across. Combining several frames should help, because turbulence jitter averages toward zero over time while the object persists. But to combine frames, a network must first shift each past frame so the object lines up with the current one, and current alignment modules cannot tell "the object moved" apart from "the air wobbled." A flexible learned warp will happily chase the wobble. When it does, it also learns the specific wobble statistics of whatever simulator it was trained on, and that is precisely what fails on real sensors.
This note proposes a simple, testable fix. At each level of the detector's feature pyramid, force the cross-frame warp to be a constant velocity multiplied by the time offset. A warp of that form can follow real object motion over a short window but has no capacity to fit zero-mean random jitter. After this constrained alignment, ordinary weighted averaging cancels the jitter. The claim is that this restriction does two things at once. It recovers small objects that single-frame detectors lose, and it transfers from synthetic to real turbulence better than unconstrained alignment, because it bakes in the one turbulence property that simulation gets right (zero mean over time) and ignores the properties it gets wrong.
Core Claim
Write the displacement of feature content at pyramid level $l$, location $x$, between the current frame and a frame $\tau$ steps earlier as $d_l(x,\tau) = \tau\, v_l(x) + \epsilon_l(x,\tau)$. Here $v_l(x)$ is the physical velocity of whatever occupies that location, and $\epsilon_l$ is the turbulence jitter, which is space-varying and roughly zero-mean over time. Unconstrained alignment predicts $d_l$ directly. This includes the 3D deformable convolution in DMAT and the optical-flow warping in FGFA. We instead predict only $v_l$ from a short window and warp by $\tau\, v_l$. This borrows the velocity parametrization that AsyncBEV uses to compensate sensor latency, a setting with no turbulence. The aligned features are then fused with similarity weights as in FGFA:
$$ \hat F_l(x) = \sum_{\tau=0}^{K-1} w_{l,\tau}(x)\, F_{l,t-\tau}\big(x - \tau\, v_l(x)\big), $$
where $F_{l,t}$ is the level-$l$ feature map at frame $t$, $K$ is the window length, and $w_{l,\tau}$ are normalized cosine-similarity weights. If the jitter is roughly independent across frames with per-frame variance $\sigma^2$, the residual misalignment variance after fusion falls roughly as $\sigma^2/K$, as in any average of independent zero-mean errors.
This leads to two falsifiable predictions. First, the gain over a single frame should grow as the ratio of jitter magnitude to object size grows, so it should be concentrated in the smallest size bins and at the finest pyramid levels. Second, the gap between constrained and unconstrained alignment should be small on synthetic test data and large on real data. That pattern is what we would see if unconstrained offsets absorb simulator-specific jitter.
Why this, and why it differs from existing work
DMAT is the closest work. It jointly trains a turbulence mitigator, which uses 3D deformable registration and a 3D Mamba U-Net, with a detector. It reports up to a 15% mAP[0.50–0.95] gain over cascaded mitigation-then-detection. However, it is evaluated quantitatively only on synthetic 256×256 Phase-to-Space (P2S) crops, real CLEAR results are qualitative only, multi-scale structure is not modeled, and tiny targets remain a failure mode. Our proposal differs in where alignment happens, at every feature-pyramid level rather than in image space. It also differs in what alignment is allowed to express, which is velocity only. Finally, it targets quantitative real-data evaluation rather than synthetic-only evaluation.
Three other works motivate the constraint but solve neighboring problems. Diffeomorphic Template Registration shows that the zero-mean assumption supports good restoration, but only for static scenes. Unsupervised Moving Object Segmentation with Atmospheric Turbulence averages RAFT flow over short windows and uses epipolar geometry to separate rigid motion from turbulence. This is close in spirit to ours, but it is non-differentiable, runs at about 0.95 FPS, and outputs masks rather than detections. ConVRT separates static content from a temporal deformation field, but it takes around 10 minutes per 25 frames and does no detection.
Two works constrain the design. Finding Tiny Faces shows that tiny objects are found mostly through their surroundings, and turbulence corrupts those surroundings. This is why we fuse at feature levels whose receptive fields include that context, rather than just around the object. Single Image Deraining documents that restoring images before detection can lower real-world mAP (on RIS with YOLOv3, 23.27 on raw rainy images versus 20.45 after JORDER). This rules out a restoration front-end as our main path and is why we never optimize pixel fidelity. The resolution-degradation literature, including LIP (learned pooling raises COCO small-object AP from 21.9 to 24.0), explains why fine pyramid levels must be preserved. Our approach complements it rather than competing: temporal fusion recovers detail lost below the feature stride, which learned pooling within one frame cannot recover.
Research Program
We will build a single-stage multi-scale detector on a standard 2D backbone with a feature pyramid. At each level we add a lightweight velocity head. The head takes the $K$ most recent feature maps at that level and outputs a two-channel velocity field. Coarser-level velocities initialize finer ones, so a tiny object inherits motion estimated where it is surrounded by context, rather than from its own few noisy pixels. Fusion follows the equation above, and the detection head is unchanged.
The main training data are P2S turbulence applied to full-resolution GOT-10k car and person sequences. This uses DMAT's generator, but we do not crop to 256×256, so that small objects keep their true pixel size. We randomize turbulence strength widely. Supervision is detection loss only.
The analysis has three parts. First, on synthetic data, where the true decomposition into object motion and jitter is known, we measure what fraction of each alignment module's offset energy tracks jitter rather than motion. This tests the mechanism directly, not just the outcome. Second, we report every metric by object size and by turbulence strength, to test the scaling prediction. Third, we measure the sim-to-real drop, real AP divided by synthetic AP at matched size bins, for each alignment variant.
Baselines
The naive reference is the same single-stage detector run frame by frame on raw turbulent video and fine-tuned on single synthetic turbulent frames, with no temporal input. It sets the floor and also tests whether temporal fusion helps at all.
The second baseline is unaligned temporal averaging. We fuse the same $K$ feature maps with the same similarity weights but with zero warp. For a static long-range camera this is the "turbulence averages out" argument implemented directly. The FGFA ablations suggest it blurs moving objects, so we will report it by velocity bin.
The third baseline is flow-guided aggregation in the style of FGFA, with RAFT flow replacing FlowNet, inserted at the same pyramid levels. On clean ImageNet VID, FGFA raised fast-motion mAP from 51.4 to 57.6. It is the natural representative of "align with dense flow."
The fourth baseline is a cascade: an off-the-shelf turbulence mitigator (DATUM or TMT, as compared in DMAT) followed by the single-frame detector. This is included because it is what practitioners would deploy.
The strongest competitor is DMAT itself. It is the only method in the file that jointly trains turbulence alignment with detection, and it reports up to a 15% mAP[0.50–0.95] improvement over cascaded systems on P2S COCO and GOT-10k. No reported real-data number was found on Lacuna for DMAT, since its CLEAR evaluation is qualitative. We will retrain DMAT on our full-resolution data with the same detector head, the same number of frames, the same turbulence randomization, and the same hyperparameter search budget. If memory forces DMAT onto tiles, we will report that constraint rather than hide it. Beating it means two things. On real data, small-object AP must be higher, with confidence intervals from per-video bootstrap that exclude zero. On synthetic data, overall AP must be no more than about one point lower, at lower memory and latency.
The ablation that isolates the key ingredient keeps our architecture fixed and replaces the velocity-times-offset warp with free per-frame offsets of identical capacity. A second ablation shares a single velocity field across pyramid levels instead of using coarse-to-fine per-level fields. If the free-offset variant matches ours on real data, the core claim is false.
Why not just…?
Why not just use DMAT, or any unconstrained deformable alignment, with heavier turbulence randomization? This is the real threat. Broad randomization might teach free offsets to ignore jitter on their own. The free-offset ablation and the retrained DMAT both receive exactly our randomization, so the experiment settles this directly. If their sim-to-real drop matches ours, the constraint adds nothing beyond efficiency, and we will say so. The jitter-energy diagnostic tells us why either way.
Why not just average frames without alignment, since turbulence is zero-mean? For stationary targets this may genuinely be enough, and we expect it to be competitive in the zero-velocity bin. The FGFA ablations report that unaligned averaging hurts moving objects. The diffeomorphic-registration work reports that averaging produces blurry templates even for static scenes. The velocity-binned results will show where this baseline breaks down.
Why not just restore the video first and then detect? The deraining benchmark shows that restoring before detecting lowered real mAP from 23.27 to 20.45 on RIS. DMAT reports cascades trailing joint training by up to 15% mAP on synthetic turbulence. ConVRT-class restorers are also orders of magnitude too slow. We still run the cascade so that the comparison is on our data.
Why not just run the single-frame detector at higher input resolution or with multi-scale testing? Upscaling enlarges the jitter along with the object, so it does not change their ratio, and MSCD notes multi-scale testing costs about 7× in inference time. We will include a 1.5× upscaled single-frame variant to check this empirically.
Proposed Experiments
The synthetic evaluation uses full-resolution P2S GOT-10k car and person sequences and the P2S COCO "CarPerson" subset. We report COCO-style AP, AP for small objects, and AP stratified by turbulence strength, together with latency and peak memory.
The real evaluation has two parts. First, we convert the DOST moving-object masks (38 real long-range turbulence videos, 1,719 frames) into boxes and evaluate class-agnostic moving-object detection by size bin. As a sanity anchor, the unsupervised segmentation method there reaches Jaccard 0.703 overall and 0.557 under severe turbulence; these are mask scores, not detection numbers. Second, we annotate boxes on a subset of CLEAR sequences, a modest effort that turns DMAT's qualitative real evaluation into a quantitative one. HeatChamber is static and only has restoration ground truth, so we use it solely to check that aligned features are temporally stable, by measuring feature-level warp error. A clean ImageNet VID run checks that the velocity constraint does not hurt ordinary video detection relative to FGFA's reported 76.3% mAP.
The decisive checks follow directly from the core claim. The constrained model must beat free offsets on real small-object AP while roughly tying on synthetic data. Its advantage over single-frame detection must grow monotonically with turbulence strength relative to object size. Its offsets must carry measurably less jitter energy on synthetic sequences where ground truth is known. If the first check fails, we drop the transfer claim and report the result as an efficiency method. If the third check fails while the first passes, the gain comes from something other than jitter rejection, and we will investigate what it is.
References
- DMAT: An End-to-End Framework for Joint Atmospheric Turbulence Mitigation and Object Detection
- Flow-Guided Feature Aggregation for Video Object Detection
- AsyncBEV: Cross-modal flow alignment in Asynchronous 3D Object Detection
- Diffeomorphic Template Registration for Atmospheric Turbulence Mitigation
- Unsupervised Moving Object Segmentation with Atmospheric Turbulence
- Temporally Consistent Atmospheric Turbulence Mitigation with Neural Representations
- Finding Tiny Faces
- Single Image Deraining: A Comprehensive Benchmark Analysis
- LIP: Local Importance-Based Pooling
- Multi-Scale Cross Distillation for Object Detection in Aerial Images