Reinforcing VLAs in Task-Agnostic World Models

RLxF @ ICML 2026 · OpenReview

This paper introduces RAW-Dream (Reinforcing VLAs in task-Agnostic World Dreams), a framework that allows robots to learn new skills entirely through "imagination" without needing to collect massive amounts of real-world data for every new task.

TL;DR

RAW-Dream enables Vision-Language-Action (VLA) models to adapt to unseen tasks by training within a task-agnostic world model using zero-shot rewards from a foundation Vision-Language Model (VLM). It breaks the "task-specific data bottleneck," allowing robots to improve their performance with as few as 1–3 human demonstrations.

What Problem Does This Paper Address?

While modern robotic policies (VLAs) are impressive, they are often "brittle"—they fail when faced with new objects or slightly different instructions. Reinforcement Learning (RL) is the standard way to fix this, but doing RL on a physical robot is slow, expensive, and potentially dangerous.

Researchers have tried using "World Models" (WMs)—AI-generated video simulators—to let robots practice in their "dreams." However, current methods have a major flaw: to build a good simulator for a task, you usually need to collect thousands of real-world videos of that specific task first. This defeats the purpose of using a simulator to save data. The authors call this the "task-specific data bottleneck."

Methods

RAW-Dream decouples the physics of the world from the specifics of the task. Its architecture consists of three main pillars:

  1. Task-Agnostic World Model: Instead of training a simulator for one task, they train a video generation model (based on WAN 2.1) on diverse "play data"—random, uncurated robot interactions. This allows the model to learn general physical priors (like how objects slide or fall) that apply to any task.
  2. Zero-Shot VLM Rewards: Rather than training a custom "success detector" for every task, RAW-Dream uses an off-the-shelf Vision-Language Model (Qwen3-VL). The VLA "imagines" a video of itself performing a task, and the VLM looks at that video and simply answers: "Did the robot succeed?"
  3. Dual-Noise Verification (DNV): Video generators can "hallucinate" (e.g., an object magically appearing in a goal state). To prevent the robot from "reward hacking" these hallucinations, RAW-Dream runs the same action sequence through the world model twice with different random noise. If the VLM says "success" on one but "failure" on the other, the rollout is discarded as unreliable.

Datasets

The authors evaluated RAW-Dream in both simulation and the real world:

  • LIBERO Benchmark: They used LIBERO-90 (90 diverse tasks) to train the general world model. They then tested the system on four held-out suites (Spatial, Object, Goal, and Long) containing tasks the world model had never seen.
  • Open X-Embodiment: Used for large-scale pre-training of the real-world world model.
  • Real-World Play Data: 4 hours of uncurated, teleoperated "play" data used to fine-tune the world model for a specific tabletop environment.

Key Insights and Ideas

  • Physics is Universal: The core "aha moment" is that physical dynamics (gravity, friction, collisions) are task-independent. A bowl moves the same way whether the instruction is "pick up the bowl" or "move the bowl left." By capturing these "physical priors," a world model can simulate tasks it has never seen.
  • Semantic Anchoring: You only need a tiny bit of task-specific data (1–3 demonstrations) to tell the robot what the task is (Supervised Fine-Tuning). Once the robot understands the semantics, it can use the world model to figure out the how through trial and error.
  • Hallucination as Uncertainty: By using Dual-Noise Verification, the authors treat the stochastic nature of diffusion models not as a bug, but as a feature to detect epistemic uncertainty in the simulation.

Key Results

  • Outperforming Online RL: On the LIBERO benchmark, RAW-Dream improved a 1-shot baseline by +8.9% and outperformed traditional online RL that used 50x more real-world data.
  • Real-World Success: On an AgileX Piper robot arm, RAW-Dream achieved a +21.7% absolute gain in success rate across four contact-rich tasks (like putting a spoon in a bowl).
  • Zero-Shot Transfer: In the "Long" task suite (the hardest category), RAW-Dream in a zero-shot world model outperformed long-horizon online RL (+15.6% vs +2.2% improvement).

Limitations and Future Work

  • Simulation Ceiling: The policy can only become as good as the world model allows. If the world model cannot accurately simulate a specific complex interaction, the RL training will plateau.
  • VLM Reasoning: While Qwen3-VL is powerful, the authors noted that VLM reward quality can degrade on tasks with complex semantic goals (the "Goal" suite in LIBERO).
  • Future Directions: The authors suggest scaling the world model's capacity and data diversity, as well as exploring "flow-based" VLA architectures to further improve efficiency.

Summary updated

Abstract

Post-training Vision-Language-Action (VLA) models via reinforcement learning (RL) in learned world models has emerged as an effective strategy to adapt to new tasks without costly real-world interactions. However, while using imagined trajectories reduces the sample complexity of policy training, existing methods still heavily rely on task-specific data to fine-tune both the world and reward models, fundamentally limiting their scalability to unseen tasks. To overcome this, we argue that world and reward models should capture transferable physical priors that enable zero-shot inference. We propose RAW-Dream (Reinforcing VLAs in task-Agnostic World Dreams), a new paradigm that completely disentangles world model learning from downstream task dependencies. RAW-Dream utilizes a world model pre-trained on diverse task-free behaviors for predicting future rollouts, and an off-the-shelf Vision-Language Model (VLM) for reward generation. Because both components are task-agnostic, VLAs can be readily finetuned for any new task entirely within this zero-shot imagination. Furthermore, to mitigate world model hallucinations, we introduce a dual-noise verification mechanism to filter out unreliable rollouts. Extensive experiments across simulation and real-world settings demonstrate consistent performance gains, proving that generalized physical priors can effectively substitute for costly task-dependent data, offering a highly scalable roadmap for VLA adaptation.
BibTeX
@inproceedings{2026reinforcing,
  title = {{Reinforcing VLAs in Task-Agnostic World Models}},
  booktitle = {RLxF @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=vbxNmCCAoB},
}
Figure