Trusting Imagined Futures: Auditing Imagined Rewards for VLA Post-Training

Zicheng Kong, Zhenbo Xu, Haoyan Zhang, Jinghan Yang, Zhenzhen Li, Yiwei Ru, Liuyu Xiang, WanTing Xu, Min Ren, Zhaofeng HearXiv 2026 · arXiv

TL;DR

Training robots using "imagined" outcomes from video world models is powerful but risky, as reward models often overestimate the quality of hallucinated successes. Trusting Imagined Futures (TIF) introduces a risk auditor that learns to identify these overestimations by comparing imagined rewards against actual execution feedback. By gating supervision selection based on this audit risk, TIF significantly improves the success rate of Vision-Language-Action (VLA) models in both simulated benchmarks and real-world robot tasks.

What Problem Does This Paper Address?

Post-training Vision-Language-Action (VLA) models often relies on imitation learning from high-quality trajectories. To get more data, researchers use video world models to "imagine" the results of candidate actions. A reward model then scores these imagined videos to select the best trajectories for training.

However, there is a major catch: Plausibility does not equal reliability. A world model might generate a video where a robot successfully places a bottle on a rack, but in reality, that same action sequence might cause the bottle to slip or collide with an obstacle. If a reward model trusts these "hallucinated" successes, the VLA policy learns bad habits. This paper addresses the challenge of reward overestimation—how to know when to trust an imagined future and when to stick with the original, known supervision.

Figure 1. A candidate may receive a higher predicted utility than Source yet exhibit poorer execution quality. TIF audits evaluation reliability before accepting a replacement.

Methods

The authors propose Trusting Imagined Futures (TIF), a framework that adds an "auditor" to the supervision selection pipeline.

  1. Multidimensional Reward Model: Instead of a single scalar score, this model predicts a utility score plus five specific dimensions of execution quality: task progress, goal correctness, manipulation quality, safety, and stability.
  2. Risk Auditor: This is the core innovation. The auditor is trained on a dataset of paired imagined videos and real executions. It learns to predict a mismatch risk—the likelihood that the reward model is overestimating a candidate's quality based on discrepancies found in the training data.
  3. Reliability Gating: During the selection process, TIF doesn't just pick the highest-scoring candidate. It only replaces the existing "Source" trajectory if two conditions are met:
  4. The candidate's predicted advantage (gain over Source) exceeds a specific margin (0.15).
  5. The candidate's audit risk is low (≤ 0.50).

If these conditions aren't met, the system conservatively retains the original Source trajectory for behavior cloning.

Figure 2. Overview of TIF. Matched executions supervise reward learning and reliability auditing. During curation, predicted utility and audit risk determine whether to replace Source. The selected real trajectories supervise VLA post-training across all valid steps.

Datasets

  • LIBERO Benchmark: The primary evaluation suite, covering 40 tasks across four categories: Spatial, Object, Goal, and Long.
  • TIF Dataset: A newly constructed dataset of 32,000 pairs of real executions and imagined videos (using Cosmos-Predict2 2B) across 8,000 initial states. This provides the ground truth needed to train the reward model and auditor.
  • Real-Robot Data: 300 physical trajectories recorded on a JAKA S5 robot across three tasks: block pushing, cup placement, and cup relocation.

Key Insights and Ideas

  • Multidimensionality Matters: Moving from a scalar reward to a 5D reward model improved Top-1 selection accuracy from 30.99% to 33.07%. The extra "diagnostic" information helps the auditor spot subtle failures like unstable grasps that a single utility score might miss.
  • Auditing the Gap: The auditor is specifically trained to catch "high-score mismatches"—cases where the reward is high but the actual execution is poor. This targeted approach achieved an AUROC of 0.77 in identifying overestimations.
  • Conservative Replacement is Better: By using the audit gate, TIF reduced the fraction of "worse" replacements (those that actually lowered utility compared to Source) from 70.6% down to 53.2%.
  • Imagination vs. Reality: The paper quantifies the "evaluation gap." Reward models using recorded execution videos achieved 50.52% Top-1 accuracy, while those using imagined videos only hit 33.07%, highlighting how much world model errors degrade reward reliability.

Key Results

  • LIBERO Success: TIF achieved a mean success rate of 76.25%, a 3.00 percentage point improvement over training on Source trajectories alone.
  • Ablation Gains: Adding the audit gate provided a 1.47 percentage point boost over using the reward model and advantage margin without auditing.
  • Real-Robot Performance: On the JAKA S5, TIF increased the "clean completion" rate (success without collisions or errors) from 58.33% to 70.00% across 60 candidate pools.
Figure 3. Real-robot candidate selection. (a) Tasks and setup. (b) Scores and selections, with advantage relative to Source C1: TIF rejects C2 and retains C1. (c,d) Recorded executions: both reach the goal, but only C2 collides with the obstacle.

Limitations and Future Work

  • Offline Dependency: Currently, the auditor requires a dataset of matched real and imagined executions to be trained. This makes it an offline method.
  • Generalization: The experiments were conducted using OpenVLA and LIBERO. The authors note that future work should explore how these auditors perform across different policy architectures and more diverse benchmarks.
  • Feedback Loops: Adapting the auditor dynamically as the world model or policy changes (online adaptation) remains an open challenge.

Summary updated

Abstract

Video world models enable candidate actions to be evaluated through imagined outcomes, but high predicted rewards can favor trajectories with poor execution quality. We introduce Trusting Imagined Futures (TIF), an offline supervision selection method for vision-language-action (VLA) post-training that audits predicted rewards using execution feedback. A multidimensional reward model predicts a utility score for candidate ranking and five dimensions of execution quality. An auditor uses reward predictions and uncertainty to identify overestimation patterns learned from paired imagined videos and real executions. TIF accepts the candidate with the highest predicted utility when its advantage over the existing trajectory exceeds a margin and its audit risk passes a threshold; otherwise, it retains the existing supervision. The retained real trajectories then supervise behavior cloning. Across four LIBERO suites and three training seeds, TIF achieves a mean success rate of 76.25%, improving over training on the existing trajectories by 3.00 percentage points with the same trajectory count and training budget. Adding audit gating yields 1.47 percentage points with the reward model and advantage margin held fixed. The gains persist against advantage ranking and logistic regression risk selection at matched replacement counts. On recorded videos from three real robot tasks, TIF increases clean completion among selected trajectories from 58.33% to 70.00% across 60 candidate pools.
BibTeX
@misc{kong2026trusting,
  author = {Zicheng Kong and Zhenbo Xu and Haoyan Zhang and Jinghan Yang and Zhenzhen Li and Yiwei Ru and Liuyu Xiang and WanTing Xu and Min Ren and Zhaofeng He},
  title = {{Trusting Imagined Futures: Auditing Imagined Rewards for VLA Post-Training}},
  year = {2026},
  eprint = {2611.04873},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04873},
}
Figure