TL;DR
Gaussian Splatting Lucas-Kanade (GS-LK) introduces a novel analytical approach to reconstruct dynamic 3D scenes from monocular video, even when the camera is nearly static. By adapting the classical Lucas-Kanade method to derive an analytical velocity field directly from the model's warp field, GS-LK achieves superior geometric accuracy and motion tracking without relying heavily on biased, data-driven priors.
The Challenge of Low-Parallax Dynamic Scenes
Reconstructing a moving 3D scene from a single 2D video presents a fundamental "chicken and egg" problem: understanding 3D motion requires knowing the scene's structure, but determining structure typically requires motion parallax (the camera moving around the object).
While 3D Gaussian Splatting (3DGS) has revolutionized static scene reconstruction, its dynamic extensions struggle in "low-parallax" scenarios where the camera is mostly stationary. Without sufficient visual cues to place Gaussians correctly in 3D space, most current methods rely on "priors"—external AI models that estimate depth or optical flow. Unfortunately, these priors are often biased toward their training data and fail to effectively regularize the underlying "warp field" (the mathematical function governing scene movement). This often leads to blurry results or physically impossible "ghosting" artifacts.
The GS-LK Approach
To address these limitations, GS-LK shifts away from purely data-driven supervision toward a principled, analytical method.
- Analytical Warp Field Velocity: Instead of using an auxiliary neural network to guess velocities, the authors derive a velocity field $v(\mathcal{G}; t)$ analytically. By applying a Taylor series expansion to the forward warp field—the function deforming the scene from a "canonical" starting state to any time $t$—they adapt the classical Lucas-Kanade method. This links the change in Gaussian parameters over time directly to the warp field's own gradients.
- Time Integration: With this analytical velocity, a numerical solver (Runge-Kutta) integrates the motion over a small time step $\Delta t$. This predicts the precise future location of every Gaussian in the subsequent frame.
- Scene Flow Regularization: The model enforces a "scene flow loss" by comparing the analytically derived motion to reference optical flow and depth. Crucially, this is decoupled into in-plane motion (optical flow) and out-of-plane motion (depth ranking), providing a much tighter constraint on the 3D trajectories of the Gaussians.
Key Insights
- Self-Consistency: By deriving velocity analytically from the warp field itself, the motion model becomes internally consistent. The model ensures the underlying 3D mathematics of the movement make sense, rather than just attempting to match a 2D picture.
- Reducing Prior Bias: Traditional depth and flow priors often "hallucinate" motion in static backgrounds to compensate for errors. GS-LK correctly distinguishes static from dynamic regions because its analytical constraints are much harder to violate.
- Improved Tractability: Directly supervising the flow field as the difference between estimated positions in consecutive frames is often unstable. The analytical derivative provides a smoother, more tractable signal for the optimization process.
Results and Evaluation
GS-LK was evaluated on five diverse dynamic scene datasets, including Dynamic Scenes, DyCheck (a challenging low-parallax benchmark), Plenoptic Video, Hypernerf, and DAVIS 2017.
The method consistently outperforms state-of-the-art dynamic Gaussian Splatting and NeRF-based methods:
- Higher Quality: On the Dynamic Scenes dataset, GS-LK achieved a PSNR of 26.34, significantly surpassing the baseline Deformable GS (24.82).
- Sharper Geometry: Qualitative results demonstrate much crisper depth maps and fewer artifacts in novel view synthesis, particularly for fast-moving objects.
- Robustness to Low Parallax: On the difficult DyCheck dataset, GS-LK showed marked improvement in reconstructing scenes with minimal camera movement—a scenario where most explicit methods fail.
Limitations and Future Work
Despite its advantages, GS-LK has a few limitations:
- SfM Dependency: Like most 3DGS methods, it is sensitive to the initial point cloud. If the initial Structure-from-Motion (SfM) pass fails to place points on a moving object, the model struggles to generate them later.
- Calibration Sensitivity: The method relies heavily on accurate camera intrinsics and extrinsics; calibration errors can distort geometry.
- Computational Cost: Calculating the pseudo-inverse of the Jacobian for the velocity field involves expensive matrix operations, potentially bottlenecking scenes with millions of Gaussians.
Future work may involve optimizing these computations or implementing pre-filtering to ignore static Gaussians, which would significantly speed up the process.
Summary updated