Abstract
Reconstructing 4D scenes from video inputs is a crucial yet challenging task. Conventional methods usually rely on the assumptions of multi-view video inputs, known camera parameters, or static scenes, all of which are typically absent under in-the-wild scenarios. In this paper, we relax all these constraints and tackle a highly ambitious but practical task, which we termed as AnyV4D: we assume only one monocular video is available without any camera parameters as input, and we aim to recover the dynamic 4D world alongside the camera poses. To this end, we introduce GFlow, a new framework that utilizes only 2D priors (depth and optical flow) to lift a video (3D) to a 4D explicit representation, entailing a flow of Gaussian splatting through space and time.
Introduction
Reconstructing a 3D world that changes over time—a "4D" reconstruction—is a holy grail of computer vision. Traditionally, this requires expensive multi-camera setups or "static scene" assumptions where only the camera moves. Doing this from a single, "in-the-wild" monocular video (like a phone clip) is incredibly difficult because it is an "ill-posed" problem: from a single 2D viewpoint, it is hard to distinguish whether an object is moving or if the camera is moving around it. Existing methods often rely on implicit representations like Neural Radiance Fields (NeRFs), which can be slow and difficult to constrain physically. GFlow aims to recover both the 4D geometry and the camera poses simultaneously from a single video stream.
Methodology
GFlow represents the world as a "flow" of 3D Gaussians. Unlike pixels or voxels, these are flexible, 3D "blobs" that can be moved and reshaped. The framework follows a clever alternating optimization strategy:
- Prior Association: It starts by using off-the-shelf foundation models to estimate 2D depth and optical flow for every frame.
- Movement Clustering: It calculates "epipolar error" to distinguish between the static background (still cluster) and moving objects (moving cluster).
- Alternating Optimization: It freezes the 3D points to optimize camera pose via static parts, then uses the new pose to update the properties of the 3D Gaussians, guided by 2D optical flow.
- Pixel-wise Densification: To handle new content appearing in the frame, GFlow identifies regions with high error and "seeds" new Gaussians there.
Key Results and Insights
The most powerful idea in GFlow is splitting the scene into "still" and "moving" parts. By using the static background to anchor the camera pose, the system treats object motion as a residual problem, making simultaneous optimization tractable.
On the DAVIS dataset, GFlow achieved a PSNR of 29.74, significantly outperforming baselines like CoDeF (24.89) and 4DGS (24.60). For pose accuracy on the MPI Sintel dataset, GFlow achieved an Absolute Trajectory Error (ATE) of 0.124, outperforming specialized tools like DROID-SLAM. Furthermore, GFlow is remarkably efficient, taking only 10 to 20 minutes to optimize a video.
Downstream Applications
Because GFlow uses explicit 3D Gaussians rather than an implicit neural function, it is naturally suited for diverse downstream tasks. Users can manipulate clusters of Gaussians to perform object-level editing, such as moving or resizing a car within a video.
Beyond reconstruction, GFlow enables:
- Unsupervised Segmentation: Moving objects are naturally separated from the background.
- Novel View Synthesis: Recovered camera poses allow for rendering the scene from entirely new perspectives.
- 3D Tracking: Points can be tracked across frames without specific prior training.
Conclusion
GFlow demonstrates that modern 2D foundation models for depth and flow are now robust enough to act as reliable constraints for lifting 2D pixels into a consistent 4D world. While performance remains dependent on the quality of these initial 2D priors, GFlow provides a versatile and powerful framework for in-the-wild video understanding and editing.
Summary updated