TL;DR
UGSDF (Urban Gaussians via Signed Distance Functions) is a novel framework that reconstructs dynamic urban scenes by combining the visual fidelity of 3D Gaussian Splatting (3DGS) with the geometric precision of Signed Distance Functions (SDFs). By leveraging 2D priors like metric depth and point tracking, it achieves state-of-the-art rendering quality without requiring expensive 3D tracklets, LiDAR data, or specialized object templates like SMPL.
What Problem Does This Paper Address?
Modeling dynamic urban environments—filled with moving cars, cyclists, and pedestrians—is a cornerstone of autonomous driving simulation and augmented reality. However, current state-of-the-art methods face a "supervision tax." They typically rely on high-fidelity LiDAR, manually annotated 3D bounding box tracklets, or category-specific templates (like SMPL for humans) to handle motion and geometry.
Furthermore, existing neural representations have inherent trade-offs. 3D Gaussian Splatting (3DGS) offers impressive real-time rendering but often produces "floaters"—stray artifacts that lack geometric consistency. Conversely, Signed Distance Fields (SDFs) provide excellent surface geometry but struggle to capture high-frequency visual details or complex dynamic deformations in sparse-view settings. UGSDF aims to bridge this gap by creating a robust, dual-representation model that works with minimal 2D supervision.
Methods
The core innovation of UGSDF is the joint optimization of 3DGS and SDF representations. The pipeline follows several key stages:
- Canonicalization via 2D Priors: Instead of 3D tracklets, the authors use off-the-shelf 2D point trackers (CoTracker) and metric depth networks (UniDepth) to lift pixels into 3D. These points are warped into a "canonical frame" to create an initial scaffold for the object.
- Dual Representation:
- 3DGS: Primitives are used for high-fidelity rendering and motion modeling. Motion is represented as a linear combination of learnable basis trajectories, allowing the model to capture complex movements efficiently.
- SDF: A deformation network (Figure 3) models the underlying geometry. It uses a multi-resolution feature grid to map observed points back to the canonical space.
- Mutual Guidance: This is the "secret sauce." The two representations help each other during training:
- SDF Guidance for 3DGS: The SDF network identifies the true surface. Gaussians far from this surface are pruned, while sparse regions near the surface are densified (Figure 4). This eliminates "floaters."
- 3DGS Guidance for SDF: The rendered depth from the Gaussians is used to narrow down the ray-sampling range for the SDF network, making surface reconstruction faster and more accurate.
Datasets
The authors evaluated UGSDF on three primary benchmarks:
- Nerf-On-The-Road (NOTR): A subset of the Waymo Open Dataset featuring complex urban scenes with diverse actors (trams, cyclists, etc.).
- KITTI MOT: A standard benchmark for multi-object tracking and reconstruction in urban environments.
- iPhone Dataset: Used to demonstrate the method's versatility on "casual" monocular videos captured without professional sensors.
Key Insights and Ideas
- The "Best of Both Worlds" Synergy: The paper demonstrates that 3DGS and SDFs are complementary. The SDF acts as a geometric "anchor" that prevents Gaussians from drifting, while the Gaussians provide the appearance and motion cues that help the SDF converge in sparse-view scenarios.
- LiDAR is Optional: One of the most striking findings is that UGSDF performs nearly as well using only 2D priors as it does with LiDAR. This significantly lowers the barrier for reconstructing scenes from standard video.
- Template-Free Modeling: Unlike many competitors that require a human template (SMPL) to reconstruct pedestrians, UGSDF treats all dynamic entities as general deformable objects, making it more flexible for atypical actors like cyclists or oddly shaped vehicles.
Key Results
- State-of-the-Art Performance: On the Waymo (NOTR) dataset, UGSDF achieved a PSNR of 30.63 for novel view synthesis, outperforming the previous leader, OmniRe (29.35), by a significant margin.
- Geometric Fidelity: Qualitative results show that UGSDF captures fine details—like a pedestrian's feet or the smooth surface of a car—much better than methods relying solely on splatting or NeRFs.
- Scene Editing: Because the model decomposes the scene into individual object nodes, it enables seamless editing tasks, such as removing a car from a street or duplicating a pedestrian in a new location.
Limitations and Future Work
- Tracking Dependency: The method is sensitive to the quality of the initial 2D tracks. If the point tracker fails (e.g., due to extreme occlusion), the reconstruction quality degrades.
- Human Expressivity: While template-free, the SDF representation currently lacks the specific prior knowledge of human anatomy that SMPL-based methods possess, which can be a disadvantage for extremely complex poses.
- View Dependency: Like most radiance field methods, rendering quality drops if the novel viewpoint deviates significantly from the original camera trajectory.
- Future Directions: The authors suggest integrating generative video priors to fill in occluded regions and exploring joint motion segmentation to further automate the pipeline.
Summary updated