GFlow: Recovering 4D World from Monocular Video

Shizun Wang, Xingyi Yang, Qiuhong Shen, Zhenxiang Jiang, Xinchao Wang

TL;DR

GFlow is a novel framework that reconstructs a complete 4D world—dynamic 3D scenes plus camera trajectories—from a single, uncalibrated monocular video. By combining 3D Gaussian Splatting with 2D priors like depth and optical flow, it enables high-fidelity novel view synthesis, object tracking, and scene editing without requiring multi-view setups or known camera parameters.

What Problem Does This Paper Address?

Reconstructing a dynamic 3D world from a single video (4D reconstruction) is a classic "ill-posed" problem: there are infinitely many 3D configurations that could project to the same 2D video. Most existing solutions "cheat" by requiring pre-calibrated camera poses, multi-view camera rigs, or by assuming the scene is entirely static.

In "the wild," we rarely have these luxuries. We usually just have a single video from a moving phone or handheld camera. Previous attempts to solve this using implicit representations (like NeRFs) often struggle with physical consistency and slow rendering speeds. GFlow aims to bridge this gap by using an explicit representation that is fast, editable, and capable of handling both camera and object motion simultaneously.

Methods

GFlow represents the video as a "flow" of 3D Gaussians moving through space and time. Its pipeline relies on three main technical pillars:

  1. Prior-Driven Initialization and Densification: Instead of starting with a random point cloud, GFlow uses 2D foundation models to estimate initial depth and optical flow. It uses edge detection (Sobel operator) to place more Gaussians in texture-rich areas and a "pixel-wise densification" strategy to add new Gaussians as new content appears in the video.
  2. Movement Clustering: The system automatically segments the scene into "still" and "moving" clusters. This is critical because only the static parts of a scene provide reliable cues for estimating the camera's movement.
  3. Alternating Optimization: GFlow performs a "dance" between two optimization steps for every frame. First, it freezes the Gaussians and optimizes the camera pose by aligning the rendered still cluster with the actual video frame. Second, it freezes the camera and optimizes the Gaussian parameters (position, scale, opacity, color) using a combination of photometric, depth, and flow losses.

To prevent the Gaussians from becoming thin "needles" (a common artifact in monocular reconstruction), the authors introduce an isotropic loss that encourages the Gaussians to maintain a more spherical shape.

The GFlow pipeline: from 2D priors to movement clustering and alternating optimization.

Datasets

The authors evaluated GFlow on two primary benchmarks:

  • DAVIS (2016/2017): A standard dataset for video object segmentation containing real-world videos with complex motion. It was used to measure reconstruction quality (PSNR, SSIM, LPIPS).
  • MPI Sintel: A synthetic dataset providing ground-truth camera poses and optical flow, used specifically to evaluate the accuracy of GFlow's camera pose estimation.

Key Insights and Ideas

  • Static/Dynamic Separation is Key: By explicitly separating the scene into still and moving parts, GFlow can solve for camera poses in dynamic environments where standard "Structure-from-Motion" (SfM) techniques typically fail.
  • Lifting 2D to 4D: The paper demonstrates that modern 2D priors (depth and flow) are now powerful enough to serve as the primary constraints for 4D reconstruction, effectively "lifting" flat pixels into a consistent 3D volume over time.
  • Explicit is Better for Editing: Because GFlow uses 3D Gaussians (an explicit representation) rather than a neural network (implicit), users can directly manipulate the scene—deleting objects, resizing them, or changing the camera path for a "dolly zoom" effect.
Figure 1: A) Given a monocular video in the wild, B) our proposed GFlow can reconstruct the underlying 4D world, i.e. the dynamic scene represented by 3D Gaussian splatting (Kerbl et al. 2023) and associated camera poses. Within GFlow, the Gaussians are split into still and moving clusters and and are further densified. C) GFlow facilitates a range of applications, including tracking objects in 2D and 3D, segmenting video objects, synthesizing new views, estimating consistent depth and video editing. We encourage readers to visit the anonymous website for more video illustrations.

Key Results

GFlow significantly outperforms existing state-of-the-art methods in reconstruction quality:

  • Reconstruction Quality: On the DAVIS dataset, GFlow achieved a PSNR of 29.74, a massive jump over baselines like CoDeF (24.89) and RoDynRF (24.79).
  • Efficiency: GFlow is remarkably fast, taking only 10 to 20 minutes to optimize a video, whereas many competing methods require over an hour.
  • Pose Accuracy: On the challenging MPI Sintel dataset, GFlow's camera pose estimation (measured by Absolute Trajectory Error) was comparable to or better than specialized pose-estimation frameworks.
Visual comparison showing GFlow's superior reconstruction of moving objects compared to other methods.

Limitations and Future Work

While GFlow is a major step forward, it has some dependencies:

  • Reliance on Priors: The quality of the 4D reconstruction is heavily dependent on the accuracy of the off-the-shelf depth and flow models used during preprocessing. If these models fail (e.g., due to extreme motion blur), GFlow's performance will degrade.
  • Hyperparameter Tuning: The authors note that because different videos have vastly different dynamics, some manual tuning of hyperparameters might be needed for the best results in specific cases.
  • Occlusions: While the movement clustering helps, extremely heavy occlusions in the static background can still complicate camera pose estimation.

Future work could involve integrating the 2D prior estimation directly into the optimization loop or improving the system's robustness to severe motion blur.

Summary updated

Abstract

Recovering 4D world from monocular video is a crucial yet challenging task. Conventional methods usually rely on the assumptions of multi-view videos, known camera parameters, or static scenes. In this paper, we relax all these constraints and tackle a highly ambitious but practical task: With only one monocular video without camera parameters, we aim to recover the dynamic 3D world alongside the camera poses. To solve this, we introduce GFlow, a new framework that utilizes only 2D priors (depth and optical flow) to lift a video to a 4D scene, as a flow of 3D Gaussians through space and time. GFlow starts by segmenting the video into still and moving parts, then alternates between optimizing camera poses and the dynamics of the 3D Gaussian points. This method ensures consistency among adjacent points and smooth transitions between frames. Since dynamic scenes always continually introduce new visual content, we present prior-driven initialization and pixel-wise densification strategy for Gaussian points to integrate new content. By combining all those techniques, GFlow transcends the boundaries of 4D recovery from causal videos; it naturally enables tracking of points and segmentation of moving objects across frames. Additionally, GFlow estimates the camera poses for each frame, enabling novel view synthesis by changing camera pose. This capability facilitates extensive scene-level or object-level editing, highlighting GFlow's versatility and effectiveness. Visit our project page at: https://littlepure2333.github.io/GFlow
BibTeX
@inproceedings{wang2025gflow,
  author = {Shizun Wang and Xingyi Yang and Qiuhong Shen and Zhenxiang Jiang and Xinchao Wang},
  title = {{GFlow: Recovering 4D World from Monocular Video}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i8.32847},
}
Figure