A single moving phone camera does not tell you how far away things are. Current systems that rebuild moving scenes from such video therefore lean on depth predicted frame by frame by large pretrained networks. Those predictions drift in scale from frame to frame, so the strongest methods trust only their ordering: which pixel is in front of which. That keeps the result stable, but it leaves surface orientation almost entirely unconstrained. This proposal starts from a basic fact of geometry. When a small flat patch moves relative to a camera, its image does not just slide. It stretches, shears and foreshortens in a way fully determined by the patch's tilt and distance. Point-like 3D primitives cannot use this signal, but flat primitives can.
We propose to represent a dynamic scene with surfels (small oriented 2D disks, the "flattened Gaussian" primitive of 2D Gaussian Splatting) and to supervise each one with the local deformation of the optical flow over its image footprint, not just the flow at its center. Each surfel predicts a plane-induced homography, the 3×3 projective warp a plane undergoes between two views, from its normal, its distance, and the relative motion between camera and surfel. We match the spatial derivative of that warp to the spatial derivative of measured flow. The constraint depends on normal over distance and on translation in the same ratio, so it is unaffected by the global scale ambiguity that ruins absolute depth supervision. It also gives monocular 4D reconstruction a direct, multi-frame-consistent handle on orientation, the quantity surfels need most and monocular video supplies least.
Core Claim
In existing monocular dynamic pipelines, motion cues enter only through point displacement. MoDGS (MoDGS) lifts depth and flow into 3D points and adds an ordinal depth loss. ProDyG (ProDyG) lifts 2D tracks into a motion scaffold. USplat4D (Uncertainty Matters in Dynamic Gaussian Splatting) observes that uncertainty along the viewing ray dwarfs uncertainty in the image plane. All three supervise each primitive with zeroth-order quantities: a position, a track, an ordering.
Our claim is that a planar primitive makes the first-order structure of image motion observable and supervisable, and that this structure pins down surfel orientation without trusting any per-frame normal or metric depth estimate. For a surfel with unit normal $n$ and plane offset $d$ in the frame-$t$ camera, and relative rigid motion $(R, \tau)$ from frame $t$ to $t'$, the induced warp is
$$H = K\left(R - \frac{\tau\, n^\top}{d}\right)K^{-1},$$
where $K$ is the camera intrinsic matrix. For a static surfel, $(R,\tau)$ comes from camera poses alone. For a dynamic surfel, it composes the camera motion with the surfel's own rotation and translation from the deformation field. Rescaling scene and translation together leaves $\tau/d$ unchanged, which is why the constraint survives scale drift.
We render, by the usual alpha blending, both a predicted flow field and its 2×2 spatial Jacobian $J^{\text{pred}}$ from the per-surfel homographies. We then penalize
$$\mathcal{L}_{\nabla F} = \sum_{p} c_p \,\big\| J^{\text{pred}}_p - \nabla F_p \big\|_1 ,$$
where $F$ is measured optical flow and $c_p$ is a flow-confidence weight from forward–backward consistency. A center-displacement flow loss sees the same constant flow over the whole footprint and is blind to $n$. This loss is not.
This is the classical planar-parallax intuition, pursued in 2D depth estimators like Deep Planar Parallax and DepthP+P. Those works fit one dominant reference plane. We apply the idea per primitive, inside a differentiable 4D representation, to non-rigid content.
Relation to the Closest Work
The three-way intersection is nearly empty, and each occupant is blocked by something our signal addresses.
SGIA uses surfels on monocular video. Its orientations come from the SMPL body template, so it only works for humans. 4DTAM uses template-free deforming surfels with an as-rigid-as-possible regularizer, but initializes normals from a hardware depth sensor. Our flow-Jacobian loss is meant to replace both the template and the sensor as the source of orientation.
The multi-view dynamic surfel methods constrain orientation through camera arrays. AT-GS uses flow-warped curvature consistency, and ST-2DGS uses HexPlane deformation plus near-binary opacity. We reuse ST-2DGS's canonical-plus-deformation structure and opacity model. We substitute temporal patch deformation for multi-view triangulation.
The static monocular surfel systems supply useful machinery but assume a rigid world. Flash-Mono and SurfSplat are examples.
From the static surfel literature we reuse three components:
- the unbiased ray–plane depth rendering of PGSR, which is the natural way to compute $d$;
- the effective-rank penalty of Hyung et al., to keep surfels from collapsing into needles;
- the same paper's sum-of-gradient-norms densification fix.
The Tension in the research file is real and motivates the design. Ordinal depth losses are scale-safe but say nothing about tilt. Per-frame normal priors such as those used by DGNS do constrain tilt, but they flicker over time. Our loss is scale-safe and orientation-bearing.
Research Program
Base system. We build a monocular dynamic surfel system on a canonical-surfel-plus-deformation design. Camera poses come from an off-the-shelf monocular estimator, with the self-calibration route of SC-4DGS as a fallback. Initialization follows MoDGS-style depth-and-flow lifting, and the ordinal depth loss is retained. The deformation field outputs per-surfel rotation as well as translation. This matters because rotation is exactly what the homography exposes, and it is otherwise weakly constrained in monocular video.
Rendering and flow targets. We extend the rasterizer to emit, per pixel, blended predicted flow and its analytic Jacobian. Each surfel's homography Jacobian at a pixel is closed-form, so this adds a small fixed per-pixel cost. Flow comes from RAFT on adjacent and skip-$k$ frame pairs. Skip pairs enlarge the effective baseline, since the $\tau n^\top/d$ term scales with translation.
Expected signal strength. We will characterize when the signal is informative. Under pure camera rotation the homography is independent of $n$, so the loss should give no gain. Gain should grow with the ratio of camera translation to depth. This is a falsifiable prediction tied to the "effective multi-view factor" of Gao et al..
Analysis axes. In textureless regions, the aperture problem drives flow confidence down, and per-frame normal priors return as a heavily down-weighted fallback. At topology changes and disocclusions, forward–backward inconsistency masks the loss. We will measure how much of the gain these masks cost.
Baselines
Naive reference. The do-nothing reference is per-frame unprojection of a monocular depth foundation model (GeoWizard, as used by MoDGS), meshed per frame. It gives a floor for temporal flicker and normal error.
Strongest monocular dynamic competitor on view synthesis. This is USplat4D, run from its released configuration on the same poses and training frames. On DyCheck at 2× resolution it reports 19.63 PSNR against 19.32 for MoSca (USplat4D). Beating it means matching that PSNR within noise while clearly improving geometry. We do not claim view synthesis as the main win.
Strongest competitor on geometry. This is DGNS (DGNS), which uses an SDF plus normal priors. It reports the lowest Chamfer and Earth Mover Distance on Dg-mesh and 39.29 PSNR on Beagle. We run it at its published settings and report wall-clock time alongside accuracy.
Most important baseline: our own pipeline with the key ingredient removed. This is a 2D-surfel variant of MoDGS: identical surfels, deformation field, ordinal loss, poses, initialization and iteration budget, with no flow-Jacobian loss. We run it in three versions:
- center-flow only;
- center-flow plus per-frame Marigold normal supervision;
- center-flow plus a PGSR-style depth–normal consistency term.
All variants get the same hyperparameter search budget, about a dozen runs per variant on a held-out tuning sequence. Beating these means lower normal error against LiDAR-derived normals on iPhone and lower Chamfer on Dg-mesh, with lower frame-to-frame normal flicker.
Upper reference on Sim4D. 4DTAM is given ground-truth RGB-D. It is not a competitor but marks how close RGB-only surfels can get. It reports 0.25 cm trajectory error on the curtain sequence.
Why not just…?
Why not just swap 3D Gaussians for 2D surfels inside MoDGS and add a monocular normal prior? This is the most serious threat, because surfels plus Marigold normals might already be good enough. The research file's Tension section explains why we doubt it: per-frame normals flicker, and supervising surfels with them makes surfels jitter and tear across time. DGNS needed an entire SDF to make normal priors behave. The ablation above settles it directly. If the normal-prior variant matches ours on normal error and temporal flicker, the contribution reduces to engineering, and we would report that.
Why not just use temporally smoothed video depth and get normals from depth-normal consistency? Depth-normal consistency only transfers whatever error the rendered depth has into the normals. Along-ray depth is precisely the poorly observed direction, as USplat4D's anisotropy analysis shows. The flow Jacobian is a separate measurement in the image plane, the direction monocular video observes well. The third ablation variant tests this. A real risk is that, at casual-video baselines, both signals are weak. The parallax-binned analysis will show where each one helps.
Why not just add USplat4D-style uncertainty graphs to regularize uncertain primitives? Graph regularization propagates motion from confident to uncertain nodes. It does not create orientation evidence where none was measured. It is complementary, so we will also report our loss stacked on a USplat4D-style graph. If the graph alone closes the geometry gap, our claim fails.
Why not just mask dynamics and run static surfel reconstruction? That discards the foreground, which is the point of 4D reconstruction. WildGS-SLAM and RobustSplat show this approach recovers the background only.
Proposed Experiments
Primary geometry test: iPhone/DyCheck. We use the 14 iPhone sequences of Gao et al., whose LiDAR depth lets us compute depth error after median scale alignment and normal angular error from LiDAR-derived normals. We also report masked PSNR, LPIPS and PCK-T for comparability. That paper documents 4–5 dB PSNR drops and roughly 30% PCK-T drops once camera teleportation is removed, so this benchmark guards against flattering setups.
Additional benchmarks. Mesh accuracy is measured with Chamfer and Earth Mover Distance on Dg-mesh. Normal and trajectory accuracy use RGB-only renderings of Sim4D, which has ground-truth normals. Extreme-viewpoint robustness at 120°–180° uses USplat4D's Objaverse benchmark. Breadth checks use DyNeRF monocular subsampling (MoDGS reports 22.64 PSNR) and NeRF-DS.
Decisive check 1: the parallax curve. We bin evaluation frame pairs by the ratio of camera translation to median depth. We predict that the normal-error gain over the no-Jacobian ablation is near zero for rotation-dominated pairs and rises monotonically with parallax. A flat curve would mean the gain comes from something other than the homography and would falsify the mechanism.
Decisive check 2: temporal stability of static background normals. We measure angular variance of normals across frames on static background surfels, against the Marigold-supervised variant. Flicker there is the specific failure the flow-Jacobian loss is meant to remove.
Honesty check: rendered views. If novel-view PSNR drops by more than a few tenths of a dB relative to the no-Jacobian ablation, we report the geometry–appearance trade-off rather than hiding it.
References
- MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors
- ProDyG: Progressive Dynamic Scene Reconstruction via Gaussian Splatting from Monocular Videos
- Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction
- Deep Planar Parallax for Monocular Depth Estimation
- DepthP+P: Metric Accurate Monocular Depth Estimation using Planar and Parallax
- Surfel-Based Gaussian Inverse Rendering for Fast and Relightable Dynamic Human Reconstruction From Monocular Videos
- 4DTAM: Non-Rigid Tracking and Mapping via Dynamic Surface Gaussians
- Adaptive and Temporally Consistent Gaussian Surfels for Multi-View Dynamic Reconstruction
- Space-time 2D Gaussian Splatting for Accurate Surface Reconstruction under Complex Dynamic Scenes
- Flash-Mono: Feed-Forward Accelerated Gaussian Splatting Monocular SLAM
- SurfSplat: Conquering Feedforward 2D Gaussian Splatting with Surface Continuity Priors
- PGSR: Planar-Based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction
- Effective Rank Analysis and Regularization for Enhanced 3D Gaussian Splatting
- DGNS: Deformable Gaussian Splatting and Dynamic Neural Surface for Monocular Dynamic 3D Reconstruction
- Self-Calibrating 4D Novel View Synthesis from Monocular Videos Using Gaussian Splatting
- Monocular Dynamic View Synthesis: A Reality Check