TL;DR
TraceFlow tackles the notoriously difficult problem of reconstructing moving, shiny objects from a single monocular video by combining deformable 2D Gaussian Splatting with real-time ray tracing. By using flat 2D surfels rather than 3D Gaussian blobs, it derives exact surface normals to trace accurate reflection rays into a learned dynamic environment representation. The result is real-time, photorealistic novel view synthesis that captures crisp, physically grounded specular highlights and near-field reflections.
What Problem Does This Paper Address?
Dynamic novel view synthesis from monocular video has made huge leaps with 3D Gaussian Splatting (3DGS), but specular (mirror-like or glossy) surfaces remain a major failure mode.
The issue boils down to physics and geometry:
- Inaccurate Normals: Standard 3DGS models scenes using fuzzy 3D ellipsoids with no explicit surfaces. Approximating surface normals from these volumes introduces errors, and because reflection ray directions are computed as $\mathbf{d}_{\text{out}} = \mathbf{d}_{\text{in}} - 2(\mathbf{d}_{\text{in}} \cdot \mathbf{n})\mathbf{n}$, even tiny normal errors get magnified, turning sharp reflections into blurred or distorted soup.
- Environment Map Bottlenecks: Prior dynamic reflection methods project rays into 2D dynamic environment maps. These maps assume infinitely distant lighting, completely failing to render near-field reflections (like an object reflecting a nearby surface) and limiting high-frequency detail due to map resolution limits.
Methods
TraceFlow reformulates the reconstruction into a hybrid rasterization–ray tracing architecture:
- Residual Material-Augmented 2DGS: Instead of 3D ellipsoids, the scene’s main geometry is represented by 2D Gaussian "surfels." Because 2D Gaussians are flat oriented disks, their surface normals are analytically well-defined ($\mathbf{t}_u \times \mathbf{t}_v$) without ad-hoc approximations. Each Gaussian is augmented with a learned specular tint parameter ($s_{\text{tint}}$). A time-conditioned MLP predicts dynamic residual offsets $(\Delta \mathbf{p}, \Delta \mathbf{s}, \Delta \mathbf{r}, \Delta o, \Delta s_{\text{tint}})$ over time.
- Temporal & Geometric Normal Supervision: The network is supervised with a geometric depth-normal consistency loss and a temporal-consistency loss using video-diffusion normal priors from NormalCrafter to curb frame-to-frame normal flickering.
- Dynamic Environment Gaussians: Rather than an equirectangular environment texture, the surrounding environment is modeled with a separate set of deformable Gaussians updated over time via a residual correction network. This naturally handles both distant environment lighting and near-field dynamic reflections.
- Hybrid Rendering Pipeline:
- Diffuse Branch: Standard 2DGS rasterization computes diffuse color ($C_{\text{diffuse}}$) and a view-dependent specular weight ($\alpha_{\text{spec}}$) via alpha blending.
- Specular Branch: Accurate reflection rays are launched from surface intersections and traced into the Dynamic Environment Gaussians using OptiX-accelerated BVH traversal to compute $C_{\text{specular}}$.
- Final pixel color is composited as $C = (1 - \alpha_{\text{spec}})C_{\text{diffuse}} + \alpha_{\text{spec}}C_{\text{specular}}$.
- Coarse-to-Fine Staged Training: Because diffuse and specular colors are unsupervised individually, early joint training tends to collapse into degenerate local minima. TraceFlow trains in three stages over 60k iterations:
- Diffuse-only (0–9k steps) to lock in baseline geometry and normals.
- Specular-only (9k–15k steps) with frozen geometry to learn environment radiance.
- Joint fine-tuning (15k–60k steps) for full physical refinement.
Datasets
The authors evaluate TraceFlow on existing public benchmarks (no new datasets were introduced):
- NeRF-DS: Seven real-world monocular dynamic video sequences featuring deforming, highly specular objects (e.g., Plate, Basin, Bell, Press).
- HyperNeRF: Used to evaluate general dynamic novel-view synthesis performance on non-specular-centric casual videos.
Key Insights and Ideas
- 2D Gaussians Solve the Normal Bottleneck: Switching from volumetric 3D Gaussians to oriented 2D surfels provides exact, stable normals off-the-shelf, eliminating the need for crude spatial-derivative approximations that ruin specular reflections.
- Explicit Geometry Over Latent Environment Maps: Representing surrounding environment lighting as 3D Gaussians rather than distant skyboxes enables real ray-object intersection, finally allowing near-field reflections (such as an apple reflected on a dynamic press plate) to resolve cleanly.
- Decoupled Training as a Regularizer: Staging the optimization (geometry first, reflections second, joint polish last) prevents the ill-posed decomposition from baking specular highlights directly into surface albedo.
Key Results
- NeRF-DS Benchmark: TraceFlow sets a new state-of-the-art across all seven dynamic specular scenes, outperforming the previous leading dynamic specular method (SpectroMotion):
- PSNR: 24.91 dB (vs. 24.17 dB, a +0.74 dB gain; reaching up to 27.39 dB on Press and 27.95 dB on Sieve).
- SSIM: 0.8880 (vs. 0.8522, a +0.0358 improvement).
- LPIPS: 0.1471 (vs. 0.1778, a 0.0307 perceptual error reduction).
- Rendering Speed: Achieves real-time novel-view synthesis at 32 FPS on an NVIDIA RTX 6000 Ada GPU (comparable to rasterization-only pipelines like SpectroMotion at 33 FPS), training in under 3 hours.
Limitations and Future Work
- Dependency on Monocular Geometry Quality: The fidelity of ray tracing is bounded by surface reconstruction quality; severe non-rigid deformations or fast motion can still produce geometric artifacts that warp reflections.
- Ray Tracing Approximations: The OptiX-accelerated ray tracer uses bounding volume hierarchy heuristics that are not fully differentiable end-to-end, introducing subtle appearance approximations.
- Future Work: Developing fully differentiable dynamic ray-splat intersection routines and leveraging stronger monocular geometric foundation models to tackle extreme non-rigid motions.
Summary updated