PhysGaia: A Physics-Aware Dataset of Multi-Body Interactions for Dynamic Novel View Synthesis

Mijeong Kim, Gunhee Kim, Wonjae Roh, Jungyoon Choi, Bohyung HanSubmission to NeurIPS 2025 Datasets and Benchmarks Track · Rejected

TL;DR

PhysGaia is a physics-aware dataset of 17 dynamic scenarios specifically designed to benchmark Dynamic Novel View Synthesis (DyNVS) on multi-body interactions across complex physical phenomena. By providing ground-truth 3D particle trajectories, physical parameters, and material-specific simulations (liquids, gases, textiles, and viscoelastic substances), it exposes how current 4D Gaussian Splatting (4DGS) methods frequently "cheat" visual appearance without learning true physical dynamics.

What Problem Does This Paper Address?

Dynamic Novel View Synthesis (DyNVS) algorithms—such as 4D Gaussian Splatting and dynamic NeRFs—aim to reconstruct 3D scenes over time from monocular or multi-view videos. While visual rendering quality has improved dramatically, existing benchmarks predominantly focus on rigid objects, single-material setups, or simple deformations (like moving humans or turning heads).

Consequently, models optimize strictly for photorealism rather than physical realism. In scenarios involving complex multi-body contact, force exchange, splashing, or turbulent flow, existing DyNVS algorithms struggle. Furthermore, because real-world video benchmarks cannot provide underlying physical ground truth, there has been no standardized way to quantitatively evaluate physical reasoning or inverse physics estimation in dynamic novel view synthesis.

Methods

To construct physically grounded, high-fidelity dynamic scenes, the authors developed a procedural data generation pipeline in SideFX Houdini 20.5:

  • Material-Specific Solvers: Rather than relying on a single simulator, each material class is simulated using its standard governing physics:
  • Liquids: FLIP (Fluid-Implicit Particle) solver to handle incompressible Navier–Stokes flow and realistic splashing.
  • Gases: Pyro grid-based solver to model buoyancy, velocity, and temperature fields.
  • Viscoelastic Materials: MPM (Material Point Method) to model solid mechanics and internal stress propagation in substances like dough, jelly, and soil.
  • Textiles: Vellum (Extended Position Based Dynamics / XPBD) combined with shape matching constraints to handle contact, collision, and wind dynamics.
  • Rendering & Setup: Scenes are path-traced via NVIDIA OptiX denoiser at $640 \times 720$ resolution with 256 samples/pixel.
  • Customizability & Evaluation Setup: Complete simulation node graphs are provided, allowing researchers to extract ground-truth particle trajectories, physical parameters (viscosity, Young's modulus, Poisson's ratio, temperature), depth maps, and surface normals. The dataset includes both multi-view camera trajectories and realistic continuous handheld monocular camera paths.

Datasets

  • PhysGaia (Proposed Benchmark): Contains 17 simulated multi-body interaction scenes covering four material types: liquid (Hanok, Ship, Cereal, Ice), gas (Box-smoke, Pisa), viscoelastic (Pancake, Jelly party, Cow, Bouncing balls), and textiles (Basin, Lucy, Tube-flag, Flags).
  • Prior Benchmarks Compared: Evaluates and contrasts against existing DyNVS datasets (e.g., Plenoptic, D-NeRF, HyperNeRF, DyCheck, NeRF-DS) and simulation-based sets (PAC-NeRF, Spring-Gaus, ScalarFlow, CLOTH4D), highlighting that existing benchmarks lack rich multi-object, multi-material force interactions.

Key Insights and Ideas

  • Visual Plausibility $\neq$ Physical Faithfulness: Because Gaussian Splatting optimizes photometric error, primitives often fail to follow actual physical motion. In gas and liquid scenes, Gaussian primitives frequently hover near surfaces and fluctuate locally rather than traveling upward with the actual fluid velocity field.
  • Current Motion Priors Struggle on Fluidity: Popular regularization methods like As-Rigid-As-Possible (ARAP) or neural motion factorizations (e.g., SOM) fail when subjected to severe topology changes, fluid splashing, or contact collisions between disparate materials.
  • Unified Testing Bed: By bundling exact physical properties alongside 3D trajectories, PhysGaia bridges DyNVS with differentiable simulation, enabling rigorous evaluation of inverse parameter estimation (e.g., inferring viscosity from video).

Key Results

The authors benchmarked leading 4DGS approaches—D-3DGS, 4DGS, STG, and SOM:

  • Sub-30 dB Performance Across All Models: Even with full multi-view training supervision, all methods failed to achieve PSNR scores above 30 dB. For example, multi-view 4DGS achieved 25.1 dB on Liquid, 24.2 dB on Gas, 21.0 dB on Viscoelastic, and 26.6 dB on Textile scenes.
  • Severe Degradation in Monocular Setups: In monocular settings, methods relying heavily on tracking priors collapsed under dynamic splashing and fast collisions; SOM dropped to 12.0 dB PSNR on viscoelastic materials.
  • Visual Artifacts: All baselines produced noticeable "needle-like" spiky Gaussians and suffered from under-reconstruction of fine-scale dynamic elements (droplets, thin-sheet folds, rising plumes).

Limitations and Future Work

  • Integration of Volumetric Physics in Particle Models: Adapting grid-based volumetric physics (such as temperature-driven buoyancy in smoke) into explicit, particle-based representations like 3D/4D Gaussians remains an unresolved theoretical and engineering challenge.
  • Sim-to-Real Domain Gap: The scenes are purely synthetic; extending physical trajectory validation to captured real-world multi-body dynamics remains difficult due to sensing limitations.
  • Future Directions: The authors suggest incorporating material-specific differentiable simulators directly into 4DGS training loops and exploring multi-physics hybrid representations capable of handling simultaneous fluid, cloth, and rigid contact.

Summary updated

Abstract

We introduce PhysGaia, a novel physics-aware dataset specifically designed for Dynamic Novel View Synthesis (DyNVS), encompassing both structured objects and unstructured physical phenomena. Unlike existing datasets that primarily focus on photorealistic reconstruction, PhysGaia is created to actively support physics-aware dynamic scene modeling. Our dataset provides complex dynamic scenarios with rich interactions among multiple objects, where they realistically collide with each other and exchange forces. Furthermore, it contains a diverse range of physical materials, such as liquid, gas, textile, and viscoelastic substances, moving beyond the rigid bodies prevalent in existing datasets. All scenes in PhysGaia are faithfully generated to strictly adhere to physical laws, leveraging carefully selected material-specific physics solvers. To enable quantitative evaluation of physical modeling, our dataset provides essential ground-truth information, including 3D particle trajectories and physics parameters, e.g., viscosity. To facilitate research adoption, we also provide essential integration pipelines for using state-of-the-art 4D Gaussian splatting models with our dataset and report their results. By addressing the critical lack of datasets for physics-aware modeling, PhysGaia will significantly advance research in dynamic view synthesis, physics-based scene understanding, and deep learning models integrated with physical simulation—ultimately enabling more faithful reconstruction and interpretation of complex dynamic scenes.
BibTeX
@inproceedings{kim2026physgaia,
  author = {Mijeong Kim and Gunhee Kim and Wonjae Roh and Jungyoon Choi and Bohyung Han},
  title = {{PhysGaia: A Physics-Aware Dataset of Multi-Body Interactions for Dynamic Novel View Synthesis}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=3pCDn5SLEo},
}
Figure