TL;DR
Training robots using "imagined" outcomes from video world models is powerful but risky, as reward models often overestimate the quality of hallucinated successes. Trusting Imagined Futures (TIF) introduces a risk auditor that learns to identify these overestimations by comparing imagined rewards against actual execution feedback. By gating supervision selection based on this audit risk, TIF significantly improves the success rate of Vision-Language-Action (VLA) models in both simulated benchmarks and real-world robot tasks.
What Problem Does This Paper Address?
Post-training Vision-Language-Action (VLA) models often relies on imitation learning from high-quality trajectories. To get more data, researchers use video world models to "imagine" the results of candidate actions. A reward model then scores these imagined videos to select the best trajectories for training.
However, there is a major catch: Plausibility does not equal reliability. A world model might generate a video where a robot successfully places a bottle on a rack, but in reality, that same action sequence might cause the bottle to slip or collide with an obstacle. If a reward model trusts these "hallucinated" successes, the VLA policy learns bad habits. This paper addresses the challenge of reward overestimation—how to know when to trust an imagined future and when to stick with the original, known supervision.
Methods
The authors propose Trusting Imagined Futures (TIF), a framework that adds an "auditor" to the supervision selection pipeline.
- Multidimensional Reward Model: Instead of a single scalar score, this model predicts a utility score plus five specific dimensions of execution quality: task progress, goal correctness, manipulation quality, safety, and stability.
- Risk Auditor: This is the core innovation. The auditor is trained on a dataset of paired imagined videos and real executions. It learns to predict a mismatch risk—the likelihood that the reward model is overestimating a candidate's quality based on discrepancies found in the training data.
- Reliability Gating: During the selection process, TIF doesn't just pick the highest-scoring candidate. It only replaces the existing "Source" trajectory if two conditions are met:
- The candidate's predicted advantage (gain over Source) exceeds a specific margin (0.15).
- The candidate's audit risk is low (≤ 0.50).
If these conditions aren't met, the system conservatively retains the original Source trajectory for behavior cloning.
Datasets
- LIBERO Benchmark: The primary evaluation suite, covering 40 tasks across four categories: Spatial, Object, Goal, and Long.
- TIF Dataset: A newly constructed dataset of 32,000 pairs of real executions and imagined videos (using Cosmos-Predict2 2B) across 8,000 initial states. This provides the ground truth needed to train the reward model and auditor.
- Real-Robot Data: 300 physical trajectories recorded on a JAKA S5 robot across three tasks: block pushing, cup placement, and cup relocation.
Key Insights and Ideas
- Multidimensionality Matters: Moving from a scalar reward to a 5D reward model improved Top-1 selection accuracy from 30.99% to 33.07%. The extra "diagnostic" information helps the auditor spot subtle failures like unstable grasps that a single utility score might miss.
- Auditing the Gap: The auditor is specifically trained to catch "high-score mismatches"—cases where the reward is high but the actual execution is poor. This targeted approach achieved an AUROC of 0.77 in identifying overestimations.
- Conservative Replacement is Better: By using the audit gate, TIF reduced the fraction of "worse" replacements (those that actually lowered utility compared to Source) from 70.6% down to 53.2%.
- Imagination vs. Reality: The paper quantifies the "evaluation gap." Reward models using recorded execution videos achieved 50.52% Top-1 accuracy, while those using imagined videos only hit 33.07%, highlighting how much world model errors degrade reward reliability.
Key Results
- LIBERO Success: TIF achieved a mean success rate of 76.25%, a 3.00 percentage point improvement over training on Source trajectories alone.
- Ablation Gains: Adding the audit gate provided a 1.47 percentage point boost over using the reward model and advantage margin without auditing.
- Real-Robot Performance: On the JAKA S5, TIF increased the "clean completion" rate (success without collisions or errors) from 58.33% to 70.00% across 60 candidate pools.
Limitations and Future Work
- Offline Dependency: Currently, the auditor requires a dataset of matched real and imagined executions to be trained. This makes it an offline method.
- Generalization: The experiments were conducted using OpenVLA and LIBERO. The authors note that future work should explore how these auditors perform across different policy architectures and more diverse benchmarks.
- Feedback Loops: Adapting the auditor dynamically as the world model or policy changes (online adaptation) remains an open challenge.
Summary updated