TL;DR
StrideWorld is a new benchmark designed to rigorously test if interactive video world models—AI that generates video in response to user commands—actually maintain a consistent 3D world. By using fixed geometric references and measuring a model's own "stride" (movement per command), it reveals that most current models suffer from significant scene deformation and fail to accurately combine movement sequences.
What Problem Does This Paper Address?
Interactive video world models (like Genie or Matrix-Game) are expected to act as "simulators" where a user can explore a 3D environment. For this to feel like a real world, two things must happen:
- Scene Rigidity: The 3D structure of the room shouldn't warp or melt as you move the camera.
- Action Consistency: Commands like "move forward" should have a stable, predictable effect over time.
Existing benchmarks often fail to catch errors because they use "fitted" references. They might reconstruct a 3D scene from the generated video itself to check for consistency. However, if the video gradually deforms, the reconstruction deforms with it, making the error invisible. Similarly, benchmarks often rescale a model's trajectory to match a ground-truth path, which can hide changes in pace or "stalling" during movement.
Methods
StrideWorld introduces a "fixed reference" approach. Instead of fitting a reference to the output, it judges the model against the ground-truth geometry of the very first frame.
The evaluation happens in two main steps:
- Camera Recovery: The researchers place 3D "anchors" at the ground-truth depth of the starting view. By tracking these points in the generated video, they recover the camera's path.
- Consistency Testing:
- Rigidity: They use "held-out" anchors (points not used to find the camera) to see if they stay in their correct 3D positions. If the scene deforms, these points will "drift."
- Pace: They measure the model’s stride—the camera motion of a single command—over the first four steps. They then test if the model maintains this exact pace for the next 12 steps.
- Composition: They test if these strides "add up" correctly when the user executes a sequence of different actions (e.g., move forward, then turn, then move left).
Datasets
The benchmark uses 140 start views across 30 diverse scenes (indoor, outdoor, natural, and built environments).
- Rendered Data: 104 views are from Unreal Engine 5, providing perfect ground-truth depth.
- Real-World Data: 36 views are from the ETH3D dataset, using laser-scan depth.
- Model Evaluation: The authors tested 15 open-source models (including Matrix-Game, HY-World, and Lyra), generating a total of 54,600 videos.
Key Insights and Ideas
- The "Absorption" Trap: Reconstructing geometry from the test video is a flawed baseline. Modern 3D foundation models are so good at hallucinating "plausible" geometry that they can fit a consistent-looking 3D model to a deforming video, effectively hiding the model's failures.
- Stride as a Universal Metric: Since different models use different units (e.g., some use keypresses, others use coordinate offsets), measuring a model's own "stride" allows for a fair comparison without needing a shared unit of motion.
- Depth-Scaling is Key: The most successful models (HY-World 2.0 and Lyra 2.0) are those that explicitly scale movement commands based on the estimated depth of the scene. This "tying" of action to geometry is crucial for consistency.
- Systematic Stalling: Many models fail not because of random noise, but because they "stall" at action switches—carrying over the previous motion or starting the new one too slowly.
Key Results
- Widespread Deformation: 14 out of 15 models failed the rigidity test, deforming their scenes by up to 7.3 times more than a rigid baseline.
- Composition Failures: While most models can keep a steady pace for a single action, they struggle to combine them. In many models, complex routes ended nowhere near the predicted destination.
- Fine-Tuning Works: The authors showed that fine-tuning models on just 112 rendered clips with exact camera paths could remove most mid-route stalls and more than double their "Composition" scores.
Limitations and Future Work
- Static Scenes Only: StrideWorld currently focuses on static environments. It does not yet account for dynamic objects like moving people or cars, which would require a more complex definition of "rigidity."
- Short Horizons: The benchmark is limited to short movement sequences (8–14 steps) where the surfaces from the starting frame are still visible. It doesn't yet evaluate long-term "loop closure" or exploration of entirely new areas.
- Planar Focus: The primary metrics focus on horizontal movement and turning (yaw), leaving full 6-degree-of-freedom (6-DoF) trajectories for future extensions.
Summary updated