TraceFlow: Dynamic 3D Reconstruction of Specular Scenes Driven by Ray Tracing

Jiachen Tao, Junyi Wu, Haoxuan Wang, Zongxin Yang, Dawen Cai, Yan YanSubmission to ICLR 2026 · Rejected

TL;DR

TraceFlow tackles the notoriously difficult problem of reconstructing moving, shiny objects from a single monocular video by combining deformable 2D Gaussian Splatting with real-time ray tracing. By using flat 2D surfels rather than 3D Gaussian blobs, it derives exact surface normals to trace accurate reflection rays into a learned dynamic environment representation. The result is real-time, photorealistic novel view synthesis that captures crisp, physically grounded specular highlights and near-field reflections.

What Problem Does This Paper Address?

Dynamic novel view synthesis from monocular video has made huge leaps with 3D Gaussian Splatting (3DGS), but specular (mirror-like or glossy) surfaces remain a major failure mode.

The issue boils down to physics and geometry:

  1. Inaccurate Normals: Standard 3DGS models scenes using fuzzy 3D ellipsoids with no explicit surfaces. Approximating surface normals from these volumes introduces errors, and because reflection ray directions are computed as $\mathbf{d}_{\text{out}} = \mathbf{d}_{\text{in}} - 2(\mathbf{d}_{\text{in}} \cdot \mathbf{n})\mathbf{n}$, even tiny normal errors get magnified, turning sharp reflections into blurred or distorted soup.
  2. Environment Map Bottlenecks: Prior dynamic reflection methods project rays into 2D dynamic environment maps. These maps assume infinitely distant lighting, completely failing to render near-field reflections (like an object reflecting a nearby surface) and limiting high-frequency detail due to map resolution limits.

Methods

TraceFlow reformulates the reconstruction into a hybrid rasterization–ray tracing architecture:

  • Residual Material-Augmented 2DGS: Instead of 3D ellipsoids, the scene’s main geometry is represented by 2D Gaussian "surfels." Because 2D Gaussians are flat oriented disks, their surface normals are analytically well-defined ($\mathbf{t}_u \times \mathbf{t}_v$) without ad-hoc approximations. Each Gaussian is augmented with a learned specular tint parameter ($s_{\text{tint}}$). A time-conditioned MLP predicts dynamic residual offsets $(\Delta \mathbf{p}, \Delta \mathbf{s}, \Delta \mathbf{r}, \Delta o, \Delta s_{\text{tint}})$ over time.
  • Temporal & Geometric Normal Supervision: The network is supervised with a geometric depth-normal consistency loss and a temporal-consistency loss using video-diffusion normal priors from NormalCrafter to curb frame-to-frame normal flickering.
  • Dynamic Environment Gaussians: Rather than an equirectangular environment texture, the surrounding environment is modeled with a separate set of deformable Gaussians updated over time via a residual correction network. This naturally handles both distant environment lighting and near-field dynamic reflections.
  • Hybrid Rendering Pipeline:
  • Diffuse Branch: Standard 2DGS rasterization computes diffuse color ($C_{\text{diffuse}}$) and a view-dependent specular weight ($\alpha_{\text{spec}}$) via alpha blending.
  • Specular Branch: Accurate reflection rays are launched from surface intersections and traced into the Dynamic Environment Gaussians using OptiX-accelerated BVH traversal to compute $C_{\text{specular}}$.
  • Final pixel color is composited as $C = (1 - \alpha_{\text{spec}})C_{\text{diffuse}} + \alpha_{\text{spec}}C_{\text{specular}}$.
  • Coarse-to-Fine Staged Training: Because diffuse and specular colors are unsupervised individually, early joint training tends to collapse into degenerate local minima. TraceFlow trains in three stages over 60k iterations:
  • Diffuse-only (0–9k steps) to lock in baseline geometry and normals.
  • Specular-only (9k–15k steps) with frozen geometry to learn environment radiance.
  • Joint fine-tuning (15k–60k steps) for full physical refinement.

Datasets

The authors evaluate TraceFlow on existing public benchmarks (no new datasets were introduced):

  • NeRF-DS: Seven real-world monocular dynamic video sequences featuring deforming, highly specular objects (e.g., Plate, Basin, Bell, Press).
  • HyperNeRF: Used to evaluate general dynamic novel-view synthesis performance on non-specular-centric casual videos.

Key Insights and Ideas

  • 2D Gaussians Solve the Normal Bottleneck: Switching from volumetric 3D Gaussians to oriented 2D surfels provides exact, stable normals off-the-shelf, eliminating the need for crude spatial-derivative approximations that ruin specular reflections.
  • Explicit Geometry Over Latent Environment Maps: Representing surrounding environment lighting as 3D Gaussians rather than distant skyboxes enables real ray-object intersection, finally allowing near-field reflections (such as an apple reflected on a dynamic press plate) to resolve cleanly.
  • Decoupled Training as a Regularizer: Staging the optimization (geometry first, reflections second, joint polish last) prevents the ill-posed decomposition from baking specular highlights directly into surface albedo.

Key Results

  • NeRF-DS Benchmark: TraceFlow sets a new state-of-the-art across all seven dynamic specular scenes, outperforming the previous leading dynamic specular method (SpectroMotion):
  • PSNR: 24.91 dB (vs. 24.17 dB, a +0.74 dB gain; reaching up to 27.39 dB on Press and 27.95 dB on Sieve).
  • SSIM: 0.8880 (vs. 0.8522, a +0.0358 improvement).
  • LPIPS: 0.1471 (vs. 0.1778, a 0.0307 perceptual error reduction).
  • Rendering Speed: Achieves real-time novel-view synthesis at 32 FPS on an NVIDIA RTX 6000 Ada GPU (comparable to rasterization-only pipelines like SpectroMotion at 33 FPS), training in under 3 hours.

Limitations and Future Work

  • Dependency on Monocular Geometry Quality: The fidelity of ray tracing is bounded by surface reconstruction quality; severe non-rigid deformations or fast motion can still produce geometric artifacts that warp reflections.
  • Ray Tracing Approximations: The OptiX-accelerated ray tracer uses bounding volume hierarchy heuristics that are not fully differentiable end-to-end, introducing subtle appearance approximations.
  • Future Work: Developing fully differentiable dynamic ray-splat intersection routines and leveraging stronger monocular geometric foundation models to tackle extreme non-rigid motions.

Summary updated

Abstract

We present TraceFlow, a novel framework for high-fidelity rendering of dynamic specular scenes by addressing two key challenges: precise reflection direction estimation and physically accurate reflection modeling. To achieve this, we propose a Residual Material-Augmented 2D Gaussian Splatting representation that models dynamic geometry and material properties, allowing accurate reflection ray computation. Furthermore, we introduce a Dynamic Environment Gaussian representation and a hybrid rendering pipeline that decomposes rendering into diffuse and specular components, enabling physically grounded specular synthesis via rasterization and ray tracing. Finally, we devise a coarse-to-fine training strategy to improve optimization stability and promote physically meaningful decomposition. Extensive experiments on dynamic scene benchmarks demonstrate that TraceFlow outperforms prior methods both quantitatively and qualitatively, producing sharper and more realistic specular reflections in complex dynamic environments.
BibTeX
@misc{tao2025traceflow,
  author = {Jiachen Tao and Junyi Wu and Haoxuan Wang and Zongxin Yang and Dawen Cai and Yan Yan},
  title = {{TraceFlow: Dynamic 3D Reconstruction of Specular Scenes Driven by Ray Tracing}},
  year = {2025},
  eprint = {2512.10095},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.10095},
}
Figure