Leveraging 2D Priors and SDF Guidance for Dynamic Urban Scene Rendering

Siddharth Tourani, Jayaram Reddy, Akash Kumbar, Satyajit Tourani, Nishant Goyal, Madhava Krishna, N. Dinesh Reddy, Muhammad Haris Khan

TL;DR

UGSDF (Urban Gaussians via Signed Distance Functions) is a novel framework that reconstructs dynamic urban scenes by combining the visual fidelity of 3D Gaussian Splatting (3DGS) with the geometric precision of Signed Distance Functions (SDFs). By leveraging 2D priors like metric depth and point tracking, it achieves state-of-the-art rendering quality without requiring expensive 3D tracklets, LiDAR data, or specialized object templates like SMPL.

What Problem Does This Paper Address?

Modeling dynamic urban environments—filled with moving cars, cyclists, and pedestrians—is a cornerstone of autonomous driving simulation and augmented reality. However, current state-of-the-art methods face a "supervision tax." They typically rely on high-fidelity LiDAR, manually annotated 3D bounding box tracklets, or category-specific templates (like SMPL for humans) to handle motion and geometry.

Furthermore, existing neural representations have inherent trade-offs. 3D Gaussian Splatting (3DGS) offers impressive real-time rendering but often produces "floaters"—stray artifacts that lack geometric consistency. Conversely, Signed Distance Fields (SDFs) provide excellent surface geometry but struggle to capture high-frequency visual details or complex dynamic deformations in sparse-view settings. UGSDF aims to bridge this gap by creating a robust, dual-representation model that works with minimal 2D supervision.

Methods

The core innovation of UGSDF is the joint optimization of 3DGS and SDF representations. The pipeline follows several key stages:

  1. Canonicalization via 2D Priors: Instead of 3D tracklets, the authors use off-the-shelf 2D point trackers (CoTracker) and metric depth networks (UniDepth) to lift pixels into 3D. These points are warped into a "canonical frame" to create an initial scaffold for the object.
  2. Dual Representation:
    • 3DGS: Primitives are used for high-fidelity rendering and motion modeling. Motion is represented as a linear combination of learnable basis trajectories, allowing the model to capture complex movements efficiently.
  3. SDF: A deformation network (Figure 3) models the underlying geometry. It uses a multi-resolution feature grid to map observed points back to the canonical space.
  4. Mutual Guidance: This is the "secret sauce." The two representations help each other during training:
  5. SDF Guidance for 3DGS: The SDF network identifies the true surface. Gaussians far from this surface are pruned, while sparse regions near the surface are densified (Figure 4). This eliminates "floaters."
    • 3DGS Guidance for SDF: The rendered depth from the Gaussians is used to narrow down the ray-sampling range for the SDF network, making surface reconstruction faster and more accurate.
Figure 2. Overview UGSDF takes posed images, dynamic object masks, 2D tracking data, and depth maps (optionally LiDAR) as input, and outputs rendered dynamic scene. The initial model is constructed through canonicalization described in Section 4.1. A dynamic 3DGS motion representation described in Section 4.2. The object of interest is represented using a combined representation of Signed Distance Functions (SDFs) and 3D Gaussian Splatting (3DGS) as detailed in Section 4.3 and the joint optimization scheme is described in Section 4.5. These two representations are jointly learned. The coarse geometry of the Gaussians constrains the ray sampling of the SDF, while SDF queries add detail to the Gaussian representation.

Datasets

The authors evaluated UGSDF on three primary benchmarks:

  • Nerf-On-The-Road (NOTR): A subset of the Waymo Open Dataset featuring complex urban scenes with diverse actors (trams, cyclists, etc.).
  • KITTI MOT: A standard benchmark for multi-object tracking and reconstruction in urban environments.
  • iPhone Dataset: Used to demonstrate the method's versatility on "casual" monocular videos captured without professional sensors.

Key Insights and Ideas

  • The "Best of Both Worlds" Synergy: The paper demonstrates that 3DGS and SDFs are complementary. The SDF acts as a geometric "anchor" that prevents Gaussians from drifting, while the Gaussians provide the appearance and motion cues that help the SDF converge in sparse-view scenarios.
  • LiDAR is Optional: One of the most striking findings is that UGSDF performs nearly as well using only 2D priors as it does with LiDAR. This significantly lowers the barrier for reconstructing scenes from standard video.
  • Template-Free Modeling: Unlike many competitors that require a human template (SMPL) to reconstruct pedestrians, UGSDF treats all dynamic entities as general deformable objects, making it more flexible for atypical actors like cyclists or oddly shaped vehicles.
Figure 5. Ablation Analysis Our ablation analysis confirms the need for dense representations of thin objects like pedestrians and cyclists (rows 2 and 3). Furthermore, removing SDF guidance for Gaussian primitive distribution adversely impacts rendering qual- ity (rows 1 and 3).

Key Results

  • State-of-the-Art Performance: On the Waymo (NOTR) dataset, UGSDF achieved a PSNR of 30.63 for novel view synthesis, outperforming the previous leader, OmniRe (29.35), by a significant margin.
  • Geometric Fidelity: Qualitative results show that UGSDF captures fine details—like a pedestrian's feet or the smooth surface of a car—much better than methods relying solely on splatting or NeRFs.
  • Scene Editing: Because the model decomposes the scene into individual object nodes, it enables seamless editing tasks, such as removing a car from a street or duplicating a pedestrian in a new location.

Limitations and Future Work

  • Tracking Dependency: The method is sensitive to the quality of the initial 2D tracks. If the point tracker fails (e.g., due to extreme occlusion), the reconstruction quality degrades.
  • Human Expressivity: While template-free, the SDF representation currently lacks the specific prior knowledge of human anatomy that SMPL-based methods possess, which can be a disadvantage for extremely complex poses.
  • View Dependency: Like most radiance field methods, rendering quality drops if the novel viewpoint deviates significantly from the original camera trajectory.
  • Future Directions: The authors suggest integrating generative video priors to fill in occluded regions and exploring joint motion segmentation to further automate the pipeline.

Summary updated

Abstract

Dynamic scene rendering and reconstruction play a crucial role in computer vision and augmented reality. Recent methods based on 3D Gaussian Splatting (3DGS), have enabled accurate modeling of dynamic urban scenes, but for urban scenes they require both camera and LiDAR data, ground-truth 3D segmentations and motion data in the form of tracklets or pre-defined object templates such as SMPL. In this work, we explore whether a combination of 2D object agnostic priors in the form of depth and point tracking coupled with a signed distance function (SDF) representation for dynamic objects can be used to relax some of these requirements. We present a novel approach that integrates Signed Distance Functions (SDFs) with 3D Gaussian Splatting (3DGS) to create a more robust object representation by harnessing the strengths of both methods. Our unified optimization framework enhances the geometric accuracy of 3D Gaussian splatting and improves deformation modeling within the SDF, resulting in a more adaptable and precise representation. We demonstrate that our method achieves state-of-the-art performance in rendering metrics even without LiDAR data on urban scenes. When incorporating LiDAR, our approach improved further in reconstructing and generating novel views across diverse object categories, without ground-truth 3D motion annotation. Additionally, our method enables various scene editing tasks, including scene decomposition, and scene composition.
BibTeX
@misc{tourani2025leveraging,
  author = {Siddharth Tourani and Jayaram Reddy and Akash Kumbar and Satyajit Tourani and Nishant Goyal and Madhava Krishna and N. Dinesh Reddy and Muhammad Haris Khan},
  title = {{Leveraging 2D Priors and SDF Guidance for Dynamic Urban Scene Rendering}},
  year = {2025},
  eprint = {2510.13381},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.13381},
}
Figure