TL;DR
Standard post-training for video world models often causes "outcome collapse," where models only generate the most probable future and ignore rare but physically possible ones. This paper introduces Sibling-paired Policy Optimization (SibPO), which uses multiple parallel generations ("siblings") to estimate outcome frequencies and creates a mathematically "proper" objective that forces the model to match the true distribution of physical outcomes.
What Problem Does This Paper Address?
Video world models are increasingly used as simulators for robotics and physical reasoning. However, a single starting frame often leaves the future ambiguous—for example, a falling cloth might hit or miss a target with different probabilities.
The current standard for improving these models is "physics post-training," where generated videos are scored against a single recorded "ground truth" future. While this makes individual videos look more realistic, it creates a major statistical flaw: to maximize its score, the model learns to always produce the most likely outcome. This is known as outcome collapse. If a simulator only ever shows the most probable event, it becomes useless for estimating risks or success rates in complex environments like robotics.
Methods
The authors propose Sibling-paired Policy Optimization (SibPO). Instead of treating each generated video in isolation, SibPO generates a group of "siblings" from the same input frame using independent noise.
The core innovation is a two-term reward function:
- Inverse-Frequency Weighting: It rewards agreement with the recorded "ground truth" future, but weights that reward by the inverse of how often the model currently produces that specific outcome (estimated from the siblings). This gives "extra credit" for matching the record when the outcome is rare.
- Sibling Correction Term: It subtracts a term based on how much the current generation agrees with its siblings.
Mathematically, this transforms the training objective from a simple linear score into a strictly proper divergence (specifically a Bregman divergence). This ensures that the objective is only maximized when the model's generated distribution exactly matches the true data distribution.
Datasets
The researchers evaluated SibPO across a wide range of physical reasoning benchmarks:
- PisaBench: A benchmark focused on the distribution of landing times for falling objects.
- Physion: A dataset of contact prediction scenarios (e.g., will the ball hit the block?).
- Kubric: A simulated environment for throws, bounces, and collisions.
- PAWBench & CaliBench: Transfer benchmarks used to test how well the model calibrates to physical probabilities it hasn't seen during training.
- Real-world Robotics: Evaluated using the RT-1 / SIMPLER protocol to see if better world models lead to better robot policy evaluation.
Key Insights and Ideas
- The "Mode" Trap: The paper proves that any reward function that only compares a generation to a single recorded future is inherently biased toward the mode (the most frequent outcome). This is a fundamental limitation of standard RLHF-style approaches when applied to ambiguous physical futures.
- Siblings as Frequency Estimators: By using a finite group of siblings, the model can estimate the "inverse frequency" of an outcome without needing to know the true underlying distribution beforehand.
- Proper Objectives Matter: Unlike "diversity rewards" which often add arbitrary noise to prevent collapse, SibPO derives its objective from the data itself. It doesn't just ask for any diversity; it asks for the correct diversity.
Key Results
SibPO consistently outperformed existing state-of-the-art methods like Flow-GRPO and DPH-RL in maintaining physical diversity:
- Better Distribution Matching: On PisaBench, SibPO reduced the Kolmogorov-Smirnov (KS) distance (a measure of how much two distributions differ) by 9.1–10.2% compared to the best non-sibling methods.
- Preserving Rare Outcomes: Qualitatively, while other models would "land" a falling object at a single consistent time, SibPO correctly maintained the spread of landing times seen in the simulator.
- Robotics Impact: When used to evaluate robot policies, SibPO lowered the error in estimated success rates by 13%.
- No Quality Trade-off: Crucially, these gains in distribution accuracy came without degrading the visual quality or physical plausibility of individual videos.
Limitations and Future Work
- Readout Dependency: SibPO requires a "readout" function (e.g., a segmentation mask or a contact detector) to categorize outcomes. While the authors show this can be done with off-the-shelf tools like SAM 2 or CoTracker, the quality of the calibration depends on the accuracy of these readouts.
- Computational Cost: Training requires generating multiple siblings per step. While the authors note the cost is equivalent to the popular GRPO method, it is still more expensive than simple supervised fine-tuning.
- Continuous Outcomes: While the theory covers continuous outcomes via kernels, most experiments focused on discrete "cells" of outcomes. Future work could explore more complex, high-dimensional continuous physical distributions.
Summary updated