When a camera moves, every pixel in the video shifts, and a vision system has to separate the part of that motion caused by the camera from the part caused by things actually moving in the scene. For decades the standard fix has been global motion compensation: fit a flat 2D warp between frames, such as an affine map or a homography (a projective warp that is exact only for flat scenes or pure camera rotation). This lines up the background, so whatever still differs after alignment is treated as real motion. The approach breaks as soon as the camera translates through a scene with depth. Near and far surfaces shift by different amounts (parallax), so the "leftover" motion fills up with false positives. Meanwhile, feed-forward 3D reconstruction has matured. These are networks that output a full 3D scene, often as a set of small colored 3D blobs called Gaussian splats, from a few images or a video in one pass taking tens to hundreds of milliseconds. They already predict depth and camera pose, which is exactly what a parallax-correct motion compensator needs.
We propose to turn the reconstruction into the motion compensator. A streaming system keeps a running 3D map of the static scene. For each new frame it predicts the camera pose, renders the map from that pose, and compares the render with the real frame. What differs is a residual that is correct under parallax. That residual does two jobs at once. It tells the expensive encoder which image patches need recomputing, while everything else reuses cached features, much as 2D-compensated sparse inference does. It also tells the system where new dynamic content lives, so moving objects get their own Gaussians, anchored at the moment they appear rather than at frame zero. The idea in two sentences: replace the homography in classical motion compensation with "render the current 3D reconstruction from the predicted camera." Then use the resulting residual both to skip computation and to separate moving objects in a feed-forward 4D reconstructor.
Core Claim
The claim is that a render-based residual is a better compensation signal than 2D warps whenever there is parallax. We also claim that feeding it back into a streaming feed-forward reconstructor improves both cost and dynamic-scene quality compared with systems that let a transformer absorb camera motion implicitly. The residual is defined per pixel $p$ as
$$r_t(p) = v_t(p)\,\big\|\phi(I_t)(p) - \phi(\hat I_t)(p)\big\|, \qquad \hat I_t = \mathcal{R}(G^{s}_{t-1}, \hat T_t),$$
where $I_t$ is the observed frame, $G^{s}_{t-1}$ is the static Gaussian map accumulated up to the previous frame, $\hat T_t$ is the predicted camera pose, $\mathcal{R}$ is the splatting renderer, $\phi$ is a shallow, fixed feature extractor chosen for some robustness to lighting, and $v_t(p)$ is a render-confidence term that is low where the map has no coverage. The rule for pixels that are uncovered and pixels that are covered but disagree is different. Uncovered pixels are marked "newly seen" and are sent to the encoder to extend the static map. Covered pixels that disagree are candidate dynamics. A candidate is confirmed as dynamic only if its disagreement cannot be explained by a small pose correction over the next few frames. This is a 3D analogue of the dominant-versus-residual flow split used for action recognition in classical work.
Relation to Prior Work
The motivating gap comes from the motion-compensation side. MotionDeltaCNN registers frames to a persistent spherical buffer with homographies and runs sparse convolutions only on changed pixels. It reports skipping up to 90% of pixel operations. Its buffer is 2D, however, so any translating camera in a deep scene makes the background look "changed." Our persistent buffer is a 3D Gaussian map, which is the natural generalization. Omnimatte and the drone and quadrotor pipelines (Lee et al., PineSORT) all rest on the same planar assumption and serve as evidence of where that assumption holds. Tekin et al. show that compensating motion before feature extraction helps 3D prediction, but they only do translational centering of a single human crop.
Three feed-forward dynamic systems handle camera motion implicitly, and these are what we must beat. StreamSplat maps frames into an orthographic canonical space and lets the Gaussian deformation field absorb camera motion, at a reported ~0.049 s per frame. This mixes platform motion with object motion and recomputes every frame over a two-frame window. Any4D factors the scene into per-frame depth and a global pose plus scene flow, but anchors motion to the first frame. BTimer predicts a frozen Gaussian scene at a target time in 150 ms, but depends on external SLAM for poses. Our design differs from all three by making compensation explicit and geometric. The camera is explained by rendering a static map, and only the unexplained part is handed to the dynamic model. DGGS is the closest static-scene relative. It flags features that do not reproject consistently as distractors and prunes them. It discards dynamics, uses calibrated poses, and runs offline. We keep and model the dynamics, and we compute consistency against a rendered map online. H3D-DGS splits motion into observable flow components and unobservable depth components. It needs per-frame optimization and calibrated multi-camera rigs, so it supplies conceptual motivation but not a method we can reuse.
Two papers act as constraints on the design. The reality check on monocular dynamic view synthesis warns that sharp renders can hide broken geometry. Our residual is only trustworthy if the static map's geometry is right, so we evaluate depth and pose, not just images. Pose-free static reconstructors such as PF3plat (0.39 s per inference), AnySplat, and MapAnything supply the backbone. They also inherit the failure of monocular depth priors on thin and reflective structure. That failure will show up as false residual, which we measure directly.
Research Program
We will build a streaming system on top of a pose-free feed-forward Gaussian predictor. We choose a MapAnything- or AnySplat-style backbone with a pose head that can run on low-resolution tokens alone. Each frame goes through four steps. First, a cheap pass predicts $\hat T_t$. Second, the static map is rendered, and $r_t$ and $v_t$ are computed. Third, only patches whose residual or uncovered area exceeds a threshold go through the full encoder, while the others reuse cached tokens aligned by the known pose. Fourth, newly seen static patches are fused into $G^s$ with a gated merge in the spirit of FreeSplat's fusion module. Confirmed dynamic patches spawn Gaussians in a separate dynamic layer. That layer has its own birth time and a short deformation model, and its opacity lifecycle is borrowed from StreamSplat.
The pose is then refined by re-rendering only the static layer. Masking dynamic content out of the pose estimate addresses the "outlier overload" failure of RANSAC-style compensation when foreground objects dominate the frame. Training is end-to-end. The losses are render losses on held-out frames plus a weak loss that pushes $r_t$ toward zero on static pixels. The patch-selection threshold is annealed so the encoder learns to work with sparse updates.
The analytic part asks when 2D compensation is sufficient. We will measure the false-residual rate of homography compensation against render compensation as a function of parallax magnitude, which is roughly camera baseline divided by the depth range. The expectation is that the homography error grows linearly with the inverse-depth spread times the baseline, because that product is the pixel displacement a single plane cannot model. This curve tells practitioners in drone and sports video where the added machinery pays off.
Baselines
The naive reference is the same backbone run densely on every frame, with no residual gating and no dynamic layer. It fixes the compute ceiling and shows what the static-scene assumption does to moving content. It shares weights, data, and training schedule with our system, so any quality or speed difference comes from compensation alone.
The classical-compensation baseline replaces our render-based residual with a homography residual. The homography is estimated from ORB features and RANSAC, as in PineSORT, and used to gate patches exactly the way MotionDeltaCNN gates pixels. It runs on the same backbone with an identical threshold search budget. This is the key ablation that isolates the proposal's ingredient. Our system beats it only if it achieves lower false-residual rates and better static-map depth at equal or lower recompute fraction on sequences with measurable parallax. On near-planar aerial footage we expect a tie, and a tie there is the honest outcome.
The strongest existing competitors are StreamSplat for online uncalibrated dynamic reconstruction and Any4D for metric 4D geometry and scene flow. We run the released models on identical sequences and resolution, on the same GPU class. If we retrain any component on our data, we give them an equal number of training steps. StreamSplat reports ~0.049 s per frame. For its image or geometry accuracy on our benchmarks, and for Any4D and BTimer accuracy on these benchmarks, no reported result was found on Lacuna, since the benchmark section of our research file carried no numbers. We will therefore measure all baselines ourselves under a shared protocol. Beating them means three things. First, matching their per-frame latency while improving masked dynamic-region render quality and depth accuracy. Second, specifically reducing error on objects that enter after the first frame, which is Any4D's documented weakness. Third, reducing camera-object entanglement, measured as spurious motion assigned to static geometry, relative to StreamSplat. DGGS is included as the reference for static-map quality under distractors. Our static layer should match it without calibrated poses.
Two further ablations remove the dynamic confirmation test, so that every covered disagreement becomes dynamic, and remove the pose refinement on the static layer only. These show whether the separation logic, and not gating alone, carries the gains.
Why not just…?
Why not just stabilize the input video with a homography and feed it to an existing reconstructor? On distant or planar footage, such as drones at altitude, this is a genuine threat. MotionDeltaCNN and the drone-trajectory work show 2D compensation is enough there. Under translation with depth spread, a single warp misaligns every plane but one, so the stabilized video carries structured false motion. The warp also removes the true camera motion that the reconstructor needs to triangulate. The parallax-sweep experiment settles the question. If the homography residual matches ours across the range of parallax magnitudes found in real data, the proposal reduces to preprocessing, and we will say so.
Why not just let a large transformer learn camera motion implicitly, as Any4D and StreamSplat do? They do learn it, but nothing forces camera motion and object motion into separate channels. StreamSplat puts both into one deformation field, and Any4D's first-frame anchoring hurts late-entering objects. Implicit handling also rules out skipping computation, because the model does not expose which regions are already explained. The late-entry and entanglement metrics test the first point, and recompute fraction at matched quality tests the second.
Why not just use DGGS-style multi-view consistency masks? Consistency masks against reference views find distractors, but DGGS then deletes them. Computing consistency pairwise against raw frames also costs more and handles disocclusion worse than rendering one accumulated map with a coverage term. We will implement a pairwise-reprojection variant of our residual and compare mask IoU and runtime directly.
Why not just prune tokens by 2D surprise, as SURGE does for video-language models? SURGE cuts redundant tokens using predictability in image space. Under a moving camera, however, the whole frame is "surprising" in 2D even when the 3D scene is unchanged. That is exactly the case render compensation handles. A surprise-based gate is included as a variant of the classical-compensation baseline.
Proposed Experiments
The research file's benchmark inventory was cut off. We therefore rely on evaluation settings named in the cited work, plus one controlled set of our own. For dynamic monocular video we adopt the evaluation practice of the reality-check paper: masked image metrics restricted to co-visible regions, plus depth and trajectory accuracy, so that sharp but geometrically wrong outputs are exposed. For aerial and moving-platform footage we use VETRA. Its low-parallax regime is where we predict parity with homography compensation. For metric scene flow we follow Any4D's evaluation setup. We also render a synthetic suite from static Gaussian scenes with inserted moving objects. In it we sweep camera baseline and depth range on a grid and add objects at controlled entry times, which gives exact ground-truth residual masks.
The metrics are image quality (PSNR and LPIPS) over dynamic and static masks separately, depth error, camera trajectory error, dynamic-mask IoU, false-residual rate on static pixels, fraction of patches recomputed, and wall-clock time per frame.
The decisive checks are three. The first is the parallax sweep: our residual's false-positive rate must stay flat while the homography's rises, and if it does not, the method fails. The second is the late-entry test: objects appearing after frame zero must be reconstructed as well as those present from the start, which Any4D's anchoring predicts it cannot do. The third is the compute-quality frontier: at the dense baseline's quality, we must recompute substantially fewer patches. MotionDeltaCNN's reported 90% skip rate for static-scene 2D video is the upper-bound intuition, and we expect less under genuine 3D camera motion. If the static map's geometry proves too noisy, for example on thin or reflective structure, the residual will be contaminated. We will report that failure by region type rather than average it away.
References
- MotionDeltaCNN: Sparse CNN Inference of Frame Differences in Moving Camera Videos with Spherical Buffers and Padded Convolutions
- Omnimatte: Associating Objects and Their Effects in Video
- Fire Detection Algorithm for a Quad-rotor using Ego-motion Compensation
- PineSORT: A Simple Online Real-time Tracking Framework for Drone Videos in Agriculture
- Direct Prediction of 3D Body Poses from Motion Compensated Sequences
- StreamSplat: Towards Online Dynamic 3D Reconstruction from Uncalibrated Video Streams
- Any4D: Unified Feed-Forward Metric 4D Reconstruction
- Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos
- Distractor-free Generalizable 3D Gaussian Splatting
- H3D-DGS: Exploring Heterogeneous 3D Motion Representation for Deformable 3D Gaussian Splatting
- Monocular Dynamic View Synthesis: A Reality Check
- PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting
- AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Views
- MapAnything: Universal Feed-Forward Metric 3D Reconstruction
- FreeSplat: Generalizable 3D Gaussian Splatting Towards Free View Synthesis of Indoor Scenes
- SURGE: Surprise-Guided Token Reduction for Efficient Video Understanding with VLMs
- VETRA: A Dataset for Vehicle Tracking in Aerial Imagery - New Challenges for Multi-Object Tracking