TL;DR
WorldRuler is a novel framework that enables generative world models to follow custom, user-defined game rules consistently. By combining an executable 3D rule graph with a latent-space attribute replacement mechanism, it ensures that rule-driven changes—like a defeated monster dropping a key—persist across save/load cycles and manifest correctly even on objects that were off-screen when the rule was triggered.
What Problem Does This Paper Address?
Most current generative world models are trained to mimic "real-world" physics and behaviors. However, video games thrive on configurable rules that often subvert reality—think of Cat Mario, where a seemingly safe platform might suddenly collapse.
The authors identify two primary hurdles in making generative models follow such custom logic:
- Rule-State Persistence: If a player defeats a boss and saves the game, the world state (the boss is dead, the loot is on the ground) and the underlying logic (the door is now unlockable) must be preserved and restored perfectly upon loading.
- Rule-to-Visual Realization: Rules often affect objects outside the current field of view. If a switch opens a gate in another room, that gate must appear open the moment the player walks in. The model needs a way to "ground" abstract rules into specific visual instances.
Methods
WorldRuler tackles these challenges through a closed-loop system that connects rule execution, video generation, and perceptual feedback.
1. Executable 3D Rule Graph
Instead of relying on a flat memory, WorldRuler uses a graph that organizes object identities, spatial information, and interaction logic. This graph tracks "triggers" (e.g., swinging a sword) and "effects" (e.g., monster defeated). Because the graph is executable, it can be serialized (saved) and restored to maintain perfect logical consistency.
2. Dynamic Object Binding
To bridge the gap between abstract rules and pixels, the system uses Dynamic Object Binding. It associates rule-specified entities (like "Gate #1") with 3D instances in the scene. This allows the model to track the state of an object even when it is occluded or off-screen.
3. Object Attribute State Replacement
This is the core technical innovation for visual generation. During the diffusion model's denoising process, WorldRuler "reads" the latent features of an object. If the rule graph says a door should be "open" but the model is generating it "closed," the Replace module injects a correction into the latent space.
To make these edits look natural, the authors use Counterfactual Reconstruction Training. They train the model on pairs of videos where the only difference is whether a specific rule was active, teaching the network how to change one attribute (like "on fire") without distorting unrelated parts of the image.
Datasets
The authors introduced RuleVerse, a large-scale dataset of rule-driven interactions.
- Source: Collected using automated pipelines in GTA-V (for open-world behavior) and Habitat (for indoor object interactions).
- Scale: 3,200 pairs (6,400 videos total).
- Uniqueness: Each pair contains a "Rule ON" and "Rule OFF" version of the same action, providing the ground truth needed for counterfactual training. It includes complex multi-step dependencies and rules that trigger off-screen.
The model was also evaluated on the existing WorldState-Bench to test its generalization to external benchmarks.
Key Insights and Ideas
- Persistence requires logic, not just memory: Simply saving a video frame isn't enough. You must save the dependency chain (e.g., "Key is held" -> "Door can open") to ensure the world remains interactive after a reload.
- First-Appearance Control: By binding rules to 3D instances before they enter the frame, WorldRuler solves the "pop-in" problem where objects might change state the moment they become visible.
- Latent Space Surgery: The attribute replacement mechanism effectively performs "surgery" on the diffusion process, separating the logical state (the "what") from the visual rendering (the "how").
Key Results
WorldRuler significantly outperformed existing baselines like State World Models (SWM) and Code World Models (CWM) across several metrics:
- Rule Success Rate (RSR): On the Habitat subset, WorldRuler achieved 64.99%, compared to SWM's 44.06%. On GTA-V, it reached 67.42% (vs. SWM's 44.19%).
- Save/Load Consistency: The system maintained a 98.17% consistency score when restoring worlds from checkpoints, proving that the joint rule-graph/state saving approach works.
- Task Completion: In multi-stage tasks (e.g., find key -> unlock door), WorldRuler's Full Completion Rate (FCR) was nearly 3x higher than baselines in some scenarios.
Limitations and Future Work
- Perception Dependency: The system relies on external modules (like SAM2 for segmentation and DA3 for geometry). Errors in these upstream components can lead to "mis-binding" rules to the wrong objects.
- Rule Complexity: While the system handles structured logic well, it may struggle with highly fluid or continuous physical interactions that aren't easily captured in a discrete rule graph.
- Inference Overhead: Running a rule graph and performing latent replacements at every denoising step adds computational cost compared to standard video generation.
Future work aims to scale these configurable rules to even more complex, multi-agent environments and improve the robustness of the perceptual feedback loop.
Summary updated