TL;DR
InertialWorld introduces a structural prior for "inertia" into video world models, ensuring that scene content is carried forward across time rather than being reconstructed from scratch at every frame. By using a hybrid Eulerian-Lagrangian memory, the model achieves significantly more stable and physically consistent long-term video predictions across robotics, gaming, and navigation domains.
What Problem Does This Paper Address?
Imagine trying to draw a flipbook where you have to redraw every single detail of a room in every frame. Even if you're a great artist, small mistakes—a slightly shifted chair, a changing wall color—will eventually accumulate, making the room look like it's "melting" or "hallucinating" over time.
This is exactly the problem with current Video World Models (VWMs). Most models treat video prediction as a sequence of independent generation tasks. They predict the next frame based on the previous one, but they don't have an explicit mechanism to "remember" that a static table should stay exactly where it is. These small reconstruction errors compound during autoregressive rollouts (where the model's own predictions become its future inputs), leading to structural collapse and unstable dynamics.
Methods
The authors propose InertialWorld, which replaces passive memory with an active Inertial Memory. The core philosophy is "carry-before-write": the model first moves existing information to where it should be in the next frame before deciding what needs to be updated.
The architecture relies on two complementary views of the world:
- Eulerian View: A dense, scene-aligned grid that preserves global evidence (like a fixed camera view).
- Lagrangian View: Persistent "carriers" that bind to local content and move through space based on predicted, action-conditioned motion.
The Prediction Loop: * Transport: The model predicts how the Lagrangian carriers should move based on the current action (e.g., a robot arm moving right). * Predict: It generates a candidate for the next state. * Gated Update: A learned gate decides, pixel-by-pixel, whether to keep the "carried" information or overwrite it with the "newly predicted" content. This makes persistence the default behavior.
Datasets
The model was evaluated across three diverse interactive domains to test its generalizability:
- RT-1 (Robot Manipulation): Tasks involving a robot arm interacting with drawers and objects.
- Counter-Strike: Global Offensive (CSGO): 3D game simulation requiring high-speed motion and complex geometry.
- RECON: Open-world embodied navigation in outdoor environments.
The authors compared InertialWorld against several strong baselines, including Vid2World, DIAMOND, and RLVR-World, using their released checkpoints.
Key Insights and Ideas
- Inertia as a Structural Prior: Instead of just hoping a loss function teaches the model to be consistent, InertialWorld builds the concept of "things stay put unless moved" directly into the neural network's architecture.
- Persistence as the Default: By using a gated update, the model must "justify" changing a part of the scene. This drastically reduces the "drift" typically seen in long-term video generation.
- Handling Occlusion: Because the carriers are persistent, the model can "remember" what was behind a robot arm even while it's occluded. When the arm moves away, the original background is still there in memory to be "carried" back into the visible frame.
- Object-Agnostic but Region-Aware: The Lagrangian carriers aren't pre-defined objects (like "cup" or "handle"); they are learned regions. This allows the model to work without needing expensive object annotations.
Key Results
InertialWorld consistently outperformed baselines across all metrics (PSNR, SSIM, LPIPS, and MSE).
- Stability: On the RT-1 dataset, InertialWorld maintained much higher image quality over 32-step rollouts compared to baselines, which showed visible structural degradation.
- Quantitative Gains: In RT-1, it improved PSNR from 21.95 (best baseline) to 24.37. In CSGO, it achieved a PSNR of 18.36 compared to DIAMOND's 17.04.
- Motion Fidelity: Using a frozen optical flow estimator (RAFT), the authors showed that InertialWorld's predicted motion much more closely matches the ground truth than other models.
Limitations and Future Work
- Determinism: The current model is deterministic. While this is great for stability, it doesn't account for the "multimodal" nature of the future (e.g., a ball could bounce left or right).
- 2D Representation: Although it simulates 3D environments, the internal representation is still largely grid-based. Moving toward explicit 3D scene representations (like NeRFs or 3D Gaussians) within the world model is a suggested next step.
- Computational Overhead: Managing and transporting Lagrangian carriers adds complexity compared to simple frame-to-frame transformers.
Summary updated