WorldRuler: Interactive World Modeling under Configurable Rules

Mingliang Zhai, Chuanhao Li, Jiaming Tan, Zhen Li, Yuwei Wu, Yunde Jia, Zian Meng, Yuanyang Yin, Kaipeng Zhang

TL;DR

WorldRuler is a novel framework that enables generative world models to follow custom, user-defined game rules consistently. By combining an executable 3D rule graph with a latent-space attribute replacement mechanism, it ensures that rule-driven changes—like a defeated monster dropping a key—persist across save/load cycles and manifest correctly even on objects that were off-screen when the rule was triggered.

What Problem Does This Paper Address?

Most current generative world models are trained to mimic "real-world" physics and behaviors. However, video games thrive on configurable rules that often subvert reality—think of Cat Mario, where a seemingly safe platform might suddenly collapse.

The authors identify two primary hurdles in making generative models follow such custom logic:

  1. Rule-State Persistence: If a player defeats a boss and saves the game, the world state (the boss is dead, the loot is on the ground) and the underlying logic (the door is now unlockable) must be preserved and restored perfectly upon loading.
  2. Rule-to-Visual Realization: Rules often affect objects outside the current field of view. If a switch opens a gate in another room, that gate must appear open the moment the player walks in. The model needs a way to "ground" abstract rules into specific visual instances.

Methods

WorldRuler tackles these challenges through a closed-loop system that connects rule execution, video generation, and perceptual feedback.

Overview of the WorldRuler framework, showing the loop between rule execution, object binding, and video generation.

1. Executable 3D Rule Graph

Instead of relying on a flat memory, WorldRuler uses a graph that organizes object identities, spatial information, and interaction logic. This graph tracks "triggers" (e.g., swinging a sword) and "effects" (e.g., monster defeated). Because the graph is executable, it can be serialized (saved) and restored to maintain perfect logical consistency.

2. Dynamic Object Binding

To bridge the gap between abstract rules and pixels, the system uses Dynamic Object Binding. It associates rule-specified entities (like "Gate #1") with 3D instances in the scene. This allows the model to track the state of an object even when it is occluded or off-screen.

3. Object Attribute State Replacement

This is the core technical innovation for visual generation. During the diffusion model's denoising process, WorldRuler "reads" the latent features of an object. If the rule graph says a door should be "open" but the model is generating it "closed," the Replace module injects a correction into the latent space.

The local attribute replacement mechanism and counterfactual training pipeline.

To make these edits look natural, the authors use Counterfactual Reconstruction Training. They train the model on pairs of videos where the only difference is whether a specific rule was active, teaching the network how to change one attribute (like "on fire") without distorting unrelated parts of the image.

Datasets

The authors introduced RuleVerse, a large-scale dataset of rule-driven interactions.

  • Source: Collected using automated pipelines in GTA-V (for open-world behavior) and Habitat (for indoor object interactions).
  • Scale: 3,200 pairs (6,400 videos total).
  • Uniqueness: Each pair contains a "Rule ON" and "Rule OFF" version of the same action, providing the ground truth needed for counterfactual training. It includes complex multi-step dependencies and rules that trigger off-screen.

The model was also evaluated on the existing WorldState-Bench to test its generalization to external benchmarks.

Key Insights and Ideas

  • Persistence requires logic, not just memory: Simply saving a video frame isn't enough. You must save the dependency chain (e.g., "Key is held" -> "Door can open") to ensure the world remains interactive after a reload.
  • First-Appearance Control: By binding rules to 3D instances before they enter the frame, WorldRuler solves the "pop-in" problem where objects might change state the moment they become visible.
  • Latent Space Surgery: The attribute replacement mechanism effectively performs "surgery" on the diffusion process, separating the logical state (the "what") from the visual rendering (the "how").
Qualitative example of persistence: saving after defeating a monster and loading to find the state preserved.

Key Results

WorldRuler significantly outperformed existing baselines like State World Models (SWM) and Code World Models (CWM) across several metrics:

  • Rule Success Rate (RSR): On the Habitat subset, WorldRuler achieved 64.99%, compared to SWM's 44.06%. On GTA-V, it reached 67.42% (vs. SWM's 44.19%).
  • Save/Load Consistency: The system maintained a 98.17% consistency score when restoring worlds from checkpoints, proving that the joint rule-graph/state saving approach works.
  • Task Completion: In multi-stage tasks (e.g., find key -> unlock door), WorldRuler's Full Completion Rate (FCR) was nearly 3x higher than baselines in some scenarios.

Limitations and Future Work

  • Perception Dependency: The system relies on external modules (like SAM2 for segmentation and DA3 for geometry). Errors in these upstream components can lead to "mis-binding" rules to the wrong objects.
  • Rule Complexity: While the system handles structured logic well, it may struggle with highly fluid or continuous physical interactions that aren't easily captured in a discrete rule graph.
  • Inference Overhead: Running a rule graph and performing latent replacements at every denoising step adds computational cost compared to standard video generation.

Future work aims to scale these configurable rules to even more complex, multi-agent environments and improve the robustness of the perceptual feedback loop.

Summary updated

Abstract

Rules govern how player actions change interactive worlds. Different rules create distinct gameplay experiences within similar scenes and action spaces. For customizable gameplay, generative world models must follow user-defined rules beyond real-world behaviors. However, realizing user-defined rules presents two challenges: Rule-State Persistence and Rule-to-Visual Realization. The former requires preserving rule definitions, dependencies, and world states across saving and loading, so that restored worlds follow the established rules. The latter requires accurately realizing rule-induced changes on designated objects and attributes, particularly when affected objects are unseen. In this paper, we propose WorldRuler, which organizes rule definitions, dependencies, and object states in an executable 3D rule graph and connects rule execution, video generation, and perceptual feedback in a closed loop. The graph supports joint saving and restoration of rules and world states, while dynamic object binding grounds rule-specified entities in visual instances and uses stored states to constrain generation from their first appearance. We further introduce object attribute state replacement and counterfactual reconstruction training, enabling accurate attribute control and compatible appearance and motion changes while limiting interference with unrelated content. We develop an automated data collection and annotation pipeline using GTA-V and Habitat to construct RuleVerse, a dataset of rule-driven interactions with detailed state and event annotations, including cases where rules take effect before affected objects become visible. Extensive experiments demonstrate the effectiveness of WorldRuler in preserving rule-state consistency across saving and loading and accurately realizing rule outcomes on designated objects and attributes.
BibTeX
@misc{zhai2026worldruler,
  author = {Mingliang Zhai and Chuanhao Li and Jiaming Tan and Zhen Li and Yuwei Wu and Yunde Jia and Zian Meng and Yuanyang Yin and Kaipeng Zhang},
  title = {{WorldRuler: Interactive World Modeling under Configurable Rules}},
  year = {2026},
  eprint = {2611.24869},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24869},
}
Figure