TL;DR
MotionCore is a modular framework that decouples physical simulation from visual rendering in video generation. By using a "latent neural simulator" to predict rigid-body trajectories and a frozen diffusion model as a visual interface, it produces videos that are both visually realistic and physically consistent, significantly outperforming general-purpose models on physics benchmarks.
What Problem Does This Paper Address?
Current state-of-the-art video generation models (like Sora, CogVideo, or MiniMax) produce stunningly realistic pixels, but they are often "physically illiterate." Because they attempt to learn the laws of physics implicitly from visual data, they frequently generate objects that pass through one another, ignore gravity, or follow nonsensical trajectories.
The authors argue that visual modeling alone cannot provide a transferable representation of physical dynamics. To fix this, we need a way to inject explicit physical reasoning into the generative process without losing the high-fidelity appearance that modern diffusion models provide.
Methods
The authors propose a "piloted mech" approach: a dynamics-learning module acts as the "pilot" deciding how the scene moves, while a pretrained video diffusion model acts as the "body" that renders those decisions into frames.
The Architecture
The framework consists of three main stages, as illustrated in the overview:
- Latent Neural Simulator: Instead of solving complex Partial Differential Equations (PDEs) in high-resolution pixel space, MotionCore uses a Latent Neural Operator (LNO). It maps observed motion into a compact latent space, evolves the physical state (position, velocity, rotation) over time, and decodes the result.
- Physics-Informed Collision Handling: To handle the "impulsive" nature of collisions (which are hard for standard neural networks to model), they treat rigid bodies as very stiff elastic objects. They use a Collision Operator that combines learned contact probabilities with closed-form impulse updates to ensure linear momentum is conserved.
- Controllable Video Interface: The predicted physical states are converted into spatially aligned control maps. These maps guide a frozen Wan2.1-VACE video diffusion model. Crucially, the generator's backbone remains fixed, ensuring that the model's visual quality isn't degraded by the physics tuning.
Datasets
- Training: The simulator was trained on rigid-body trajectories from MOVi-Spheres (a dataset of simulated spheres with varying mass and friction) and Billiards.
- Evaluation: The authors tested the full pipeline on two major benchmarks:
- Physics-IQ-Verified: A subset of 81 scenes focusing on collision, rolling, sliding, and free fall.
- RigidBench: A public benchmark specifically designed to evaluate rigid-body dynamics in video models.
Key Insights and Ideas
- Decoupling Dynamics from Appearance: By separating the "what moves where" from the "what it looks like," the model can leverage specialized PDE solvers for motion while using general-purpose foundation models for rendering.
- Latent Factorization Theory: The paper provides a theoretical boundary (Theorem 4.1) proving that the error in latent simulation is controlled by how well the latent tokens preserve the underlying physical evolution. This justifies their use of a compressed latent space for efficiency.
- The "Residual" Advantage: The model predicts corrections to a free-motion proposal. This allows the system to rely on simple physics priors for easy movements and focus its neural capacity on complex interactions like collisions.
Key Results
MotionCore demonstrates a significant leap in physical accuracy over much larger general-purpose models:
- Physical Plausibility: On the Physics-IQ-Verified subset, MotionCore achieved a Spatio-Temporal IoU (ST-IoU) of 0.373, outperforming models like CogVideoX-5B (0.297) and Wan2.2-TI2V (0.165).
- Trajectory Obedience: On RigidBench, it achieved a mask-centroid error (L2) of 0.133, compared to 0.198 for MiniMax-H3 and 0.222 for CogVideoX.
- Qualitative Consistency: As shown in Figure 2, MotionCore successfully maintains object persistence and correct collision timing in scenarios where baseline models often cause objects to disappear or overlap.
Limitations and Future Work
Despite its strengths, the authors acknowledge several hurdles:
- State Estimation: The model's accuracy is highly dependent on the initial state estimated from the first few frames. If the initial geometry or velocity estimation is wrong, the physical rollout will be incorrect.
- Object Persistence: While better than baselines, the diffusion model can still occasionally "forget" object details over long sequences.
- Scope: The current work is specialized for rigid-body dynamics. Extending this framework to handle fluids, cloth, or articulated characters (like humans) remains a significant challenge for future research.
Summary updated