Phy124: Fast Physics-Driven 4D Content Generation from a Single Image

Jiajing Lin, Zhenzhong Wang, Yongjie Hou, Yuzhou Tang, Min JiangarXiv 2024 · OpenReview

Abstract

4D content generation focuses on creating dynamic 3D objects that change over time. Existing methods primarily rely on pre-trained video diffusion models, utilizing sampling processes or reference videos. However, these approaches face significant challenges. Firstly, the generated 4D content often fails to adhere to real-world physics since video diffusion models do not incorporate physical priors. Secondly, the extensive sampling process and the large number of parameters in diffusion models result in exceedingly time-consuming generation processes. To address these issues, we introduce Phy124, a novel, fast, and physics-driven method for controllable 4D content generation from a single image. Phy124 integrates physical simulation directly into the 4D generation process, ensuring that the resulting 4D content adheres to natural physical laws. Phy124 also eliminates the use of diffusion models during the 4D dynamics generation phase, significantly speeding up the process. Phy124 allows for the control of 4D dynamics, including movement speed and direction, by manipulating external forces. Extensive experiments demonstrate that Phy124 generates high-fidelity 4D content with significantly reduced inference times, achieving state-of-the-art performance.

Introduction

Generating 4D content—3D objects that move and deform over time—from a single 2D image is an "ill-posed" problem because the source image lacks both spatial depth and temporal motion data. Current state-of-the-art methods typically rely on video diffusion models and Score Distillation Sampling (SDS). While these produce visually interesting results, they suffer from physical inconsistency, extreme latency (often taking hours), and a lack of precise control over motion.

Phy124 addresses these bottlenecks by replacing slow, "physics-blind" video diffusion models with a Material Point Method (MPM) simulator. This shift allows for the generation of physically realistic 4D content in under 40 seconds while providing users with direct control over object dynamics.

The Phy124 Framework

Phy124 introduces a two-stage pipeline that shifts the burden of motion from neural networks to classical physics simulation.

The Phy124 pipeline, integrating 3D Gaussian Splatting with an MPM physics simulator.
  1. 3D Gaussian Splatting (3DGS) Generation: The model first converts a single input image into a static 3D representation using 3D Gaussian Splatting. This represents the object as a collection of millions of tiny, anisotropic "blobs" (Gaussians) that can be rendered very quickly.
  2. Physics-Driven 4D Dynamics: Instead of using a video model to predict the next frame, Phy124 treats each 3D Gaussian as a physical particle. It employs the Material Point Method (MPM)—a powerful simulation technique used in visual effects—to calculate how these particles move based on physical laws.
  3. External Force Control: Because it uses a real physics engine, users can apply external forces. By adjusting a simple vector, a user can control the direction and intensity of the animation, such as simulating wind or a physical interaction.

Performance and Physical Realism

By grounding the animation in the Material Point Method, the motion is mathematically guaranteed to be physically plausible, preventing the "melting" or "ghosting" effects seen in pure AI approaches. Furthermore, decoupling the spatial generation from the temporal dynamics allows for massive efficiency gains.

Comparison of generation quality and speed between Phy124 and baselines.

Key Results

  • Speed: Phy124 is remarkably fast, averaging 39.5 seconds for a full generation (approx. 24s for the 3D model and 15s for the physics simulation). In comparison, the baseline Animate124 takes roughly 9 hours (32,433 seconds).
  • Visual Quality: It achieved higher CLIP-T scores (0.9962) than both Animate124 and DG4D, indicating better consistency and alignment with the source material across different viewing angles.
  • Controllability: The framework allows for fine-grained control, such as making a sunflower sway with different intensities simply by tuning input force parameters.

Conclusion

Phy124 demonstrates that generative models do not necessarily need to handle the temporal dimension if a robust physical prior is available. While the system's output quality is currently dependent on the initial 3D reconstruction, its plug-and-play architecture allows it to benefit from future improvements in 3D generation. Future work will focus on automatically inferring material properties (like stiffness or weight) directly from images to further automate the physics-driven pipeline.

Summary updated

Abstract

4D content generation focuses on creating dynamic 3D objects that change over time. Existing methods primarily rely on pre-trained video diffusion models, utilizing sampling processes or reference videos. However, these approaches face significant challenges. Firstly, the generated 4D content often fails to adhere to real-world physics since video diffusion models do not incorporate physical priors. Secondly, the extensive sampling process and the large number of parameters in diffusion models result in exceedingly time-consuming generation processes. To address these issues, we introduce Phy124, a novel, fast, and physics-driven method for controllable 4D content generation from a single image. Phy124 integrates physical simulation directly into the 4D generation process, ensuring that the resulting 4D content adheres to natural physical laws. Phy124 also eliminates the use of diffusion models during the 4D dynamics generation phase, significantly speeding up the process. Phy124 allows for the control of 4D dynamics, including movement speed and direction, by manipulating external forces. Extensive experiments demonstrate that Phy124 generates high-fidelity 4D content with significantly reduced inference times, achieving stateof-the-art performance. The code and generated 4D content are available at the provided link: https://anonymous.4open.science/r/BBF2/.
BibTeX
@misc{lin2024phy,
  author = {Jiajing Lin and Zhenzhong Wang and Yongjie Hou and Yuzhou Tang and Min Jiang},
  title = {{Phy124: Fast Physics-Driven 4D Content Generation from a Single Image}},
  year = {2024},
  eprint = {2409.07179},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.07179},
}
Figure