TL;DR
PhysGaia is a physics-aware dataset of 17 dynamic scenarios specifically designed to benchmark Dynamic Novel View Synthesis (DyNVS) on multi-body interactions across complex physical phenomena. By providing ground-truth 3D particle trajectories, physical parameters, and material-specific simulations (liquids, gases, textiles, and viscoelastic substances), it exposes how current 4D Gaussian Splatting (4DGS) methods frequently "cheat" visual appearance without learning true physical dynamics.
What Problem Does This Paper Address?
Dynamic Novel View Synthesis (DyNVS) algorithms—such as 4D Gaussian Splatting and dynamic NeRFs—aim to reconstruct 3D scenes over time from monocular or multi-view videos. While visual rendering quality has improved dramatically, existing benchmarks predominantly focus on rigid objects, single-material setups, or simple deformations (like moving humans or turning heads).
Consequently, models optimize strictly for photorealism rather than physical realism. In scenarios involving complex multi-body contact, force exchange, splashing, or turbulent flow, existing DyNVS algorithms struggle. Furthermore, because real-world video benchmarks cannot provide underlying physical ground truth, there has been no standardized way to quantitatively evaluate physical reasoning or inverse physics estimation in dynamic novel view synthesis.
Methods
To construct physically grounded, high-fidelity dynamic scenes, the authors developed a procedural data generation pipeline in SideFX Houdini 20.5:
- Material-Specific Solvers: Rather than relying on a single simulator, each material class is simulated using its standard governing physics:
- Liquids: FLIP (Fluid-Implicit Particle) solver to handle incompressible Navier–Stokes flow and realistic splashing.
- Gases: Pyro grid-based solver to model buoyancy, velocity, and temperature fields.
- Viscoelastic Materials: MPM (Material Point Method) to model solid mechanics and internal stress propagation in substances like dough, jelly, and soil.
- Textiles: Vellum (Extended Position Based Dynamics / XPBD) combined with shape matching constraints to handle contact, collision, and wind dynamics.
- Rendering & Setup: Scenes are path-traced via NVIDIA OptiX denoiser at $640 \times 720$ resolution with 256 samples/pixel.
- Customizability & Evaluation Setup: Complete simulation node graphs are provided, allowing researchers to extract ground-truth particle trajectories, physical parameters (viscosity, Young's modulus, Poisson's ratio, temperature), depth maps, and surface normals. The dataset includes both multi-view camera trajectories and realistic continuous handheld monocular camera paths.
Datasets
- PhysGaia (Proposed Benchmark): Contains 17 simulated multi-body interaction scenes covering four material types: liquid (Hanok, Ship, Cereal, Ice), gas (Box-smoke, Pisa), viscoelastic (Pancake, Jelly party, Cow, Bouncing balls), and textiles (Basin, Lucy, Tube-flag, Flags).
- Prior Benchmarks Compared: Evaluates and contrasts against existing DyNVS datasets (e.g., Plenoptic, D-NeRF, HyperNeRF, DyCheck, NeRF-DS) and simulation-based sets (PAC-NeRF, Spring-Gaus, ScalarFlow, CLOTH4D), highlighting that existing benchmarks lack rich multi-object, multi-material force interactions.
Key Insights and Ideas
- Visual Plausibility $\neq$ Physical Faithfulness: Because Gaussian Splatting optimizes photometric error, primitives often fail to follow actual physical motion. In gas and liquid scenes, Gaussian primitives frequently hover near surfaces and fluctuate locally rather than traveling upward with the actual fluid velocity field.
- Current Motion Priors Struggle on Fluidity: Popular regularization methods like As-Rigid-As-Possible (ARAP) or neural motion factorizations (e.g., SOM) fail when subjected to severe topology changes, fluid splashing, or contact collisions between disparate materials.
- Unified Testing Bed: By bundling exact physical properties alongside 3D trajectories, PhysGaia bridges DyNVS with differentiable simulation, enabling rigorous evaluation of inverse parameter estimation (e.g., inferring viscosity from video).
Key Results
The authors benchmarked leading 4DGS approaches—D-3DGS, 4DGS, STG, and SOM:
- Sub-30 dB Performance Across All Models: Even with full multi-view training supervision, all methods failed to achieve PSNR scores above 30 dB. For example, multi-view 4DGS achieved 25.1 dB on Liquid, 24.2 dB on Gas, 21.0 dB on Viscoelastic, and 26.6 dB on Textile scenes.
- Severe Degradation in Monocular Setups: In monocular settings, methods relying heavily on tracking priors collapsed under dynamic splashing and fast collisions; SOM dropped to 12.0 dB PSNR on viscoelastic materials.
- Visual Artifacts: All baselines produced noticeable "needle-like" spiky Gaussians and suffered from under-reconstruction of fine-scale dynamic elements (droplets, thin-sheet folds, rising plumes).
Limitations and Future Work
- Integration of Volumetric Physics in Particle Models: Adapting grid-based volumetric physics (such as temperature-driven buoyancy in smoke) into explicit, particle-based representations like 3D/4D Gaussians remains an unresolved theoretical and engineering challenge.
- Sim-to-Real Domain Gap: The scenes are purely synthetic; extending physical trajectory validation to captured real-world multi-body dynamics remains difficult due to sensing limitations.
- Future Directions: The authors suggest incorporating material-specific differentiable simulators directly into 4DGS training loops and exploring multi-physics hybrid representations capable of handling simultaneous fluid, cloth, and rigid contact.
Summary updated