Abstract
Gaussian Splatting Lucas-Kanade (GS-LK) introduces a novel framework for reconstructing highly dynamic 3D scenes from monocular video. By adapting the classic Lucas-Kanade optical flow method to 3D Gaussian Splatting, the authors derive an analytical velocity field directly from a deformation network. This approach replaces "black-box" data-driven priors with precise motion constraints, resulting in more accurate geometries and smoother motion trajectories.
Introduction
Reconstructing dynamic 3D scenes from a single moving camera is a fundamental challenge in computer vision. While standard methods like 3D Gaussian Splatting (3DGS) perform well under high camera motion (motion parallax), they often struggle with static cameras or rapid object movement. Existing solutions frequently rely on external "priors"—models that estimate depth or optical flow—which can be biased and physically inconsistent, leading to "floater" artifacts and jittery motion. GS-LK addresses this by enforcing realistic motion and geometry through analytical constraints rather than over-reliance on external models.
Methods
The GS-LK framework bridges classical computer vision and modern neural representations through several key innovations. First, it utilizes Analytical Warp Fields. While most dynamic Gaussian Splatting methods rely on a Multi-Layer Perceptron (MLP) to predict simple coordinate offsets, GS-LK derives an analytical velocity field directly from this network.
Second, the authors introduce a 3D Lucas-Kanade Adaptation. Originally designed for 2D image alignment, the Lucas-Kanade method is extended here to 3D space. By calculating the Jacobian of the warp field—which describes how the output changes relative to the input—the system can determine the exact velocity of every Gaussian at any point in time. Third, the framework employs Time Integration via a numerical Runge-Kutta solver. This allows the model to integrate velocities over time, predicting future positions based on continuous trajectories rather than treating frames as isolated snapshots. Finally, Decoupled Regularization splits motion constraints into in-plane (optical flow) and out-of-plane (depth) components, enabling more effective use of specific 2D and 1D priors to refine the 3D warp field.
Key Insights
The core strength of GS-LK lies in its preference for physics over heuristics. By deriving motion analytically, the model is constrained by mathematical movement principles, preventing it from generating "fake" geometries to satisfy loss functions. The use of the Jacobian as a bridge between canonical and dynamic scenes makes the learning process more stable and tractable. Furthermore, by treating video as a continuous flow rather than discrete slides, the model produces significantly smoother trajectories, as evidenced in complex scenes like "balloon waving."
Key Results
GS-LK was evaluated on challenging real-world and synthetic datasets, including DyCheck, DAVIS 2017, and Hypernerf. The results highlight several key advantages:
- Superior Geometry: GS-LK produces significantly "crisper" depth maps compared to baselines like Deformable GS or 4DGaussians. In the "Playground" scene, it achieved a PSNR of 26.34, a notable improvement over the baseline's 24.82.
- Motion Separation: The model is exceptionally adept at distinguishing static backgrounds from dynamic foregrounds. In qualitative tests, it correctly identified stationary human subjects while accurately tracking moving objects like balloons, whereas baseline models often blurred motion across both.
- Low Parallax Performance: The method particularly shines in "near-static" camera setups. In these scenarios, traditional 3DGS often fails due to a lack of depth cues from motion parallax, but GS-LK's analytical constraints provide the necessary stability.
Limitations and Future Work
Despite its successes, GS-LK remains dependent on the initial point cloud from Structure-from-Motion (SfM). Sparse or missing points in dynamic regions can hinder reconstruction. Additionally, the computational cost of calculating pseudo-inverses for large matrices poses a scaling challenge. Future research will focus on optimizing these matrix operations and improving initialization in regions where SfM is unreliable.
Summary updated