TL;DR
Prompt-Driven Exploration (PDE) overcomes the "exploration bottleneck" in Reinforcement Learning by perturbing the natural language prompts given to foundation models rather than jittering their individual actions. By using a Vision-Language Model (VLM) to analyze failed attempts and rewrite instructions, PDE can bootstrap learning even when a policy starts with a 0% success rate, significantly improving sample efficiency in both robotics and reasoning tasks.
What Problem Does This Paper Address?
In Reinforcement Learning (RL), a policy can only improve if it discovers behaviors that yield higher rewards than its current favorites. Traditionally, we encourage this by adding "noise" to the action space—essentially making the robot's arm jitter or the agent's choices slightly stochastic.
However, for complex foundation models like Vision-Language-Action (VLA) models, this "local" jitter is often useless. If a robot tasked to "close the microwave" mistakenly tries to grab a nearby mug, small jitters won't help it realize it should be pushing the door instead. This is the "exploration bottleneck": if the initial policy is weak and never sees a single success, there is no reward signal to learn from. The authors tackle this by asking: Instead of changing the actions, what if we change the instructions?
Methods
The authors introduce Prompt-Driven Exploration (PDE), which treats natural language prompts as a high-level axis for exploration. The method operates in two main stages:
-
Update Prompt Posterior (Discovery): The system maintains a distribution of prompts. A "supervisor" VLM (like Qwen3-VL) watches videos of the policy's failed rollouts, diagnoses the failure (e.g., "the arm is biased toward the mug"), and rewrites the prompt to elicit a different global behavior (e.g., "push on the microwave door until it shuts"). This effectively performs Posterior Sampling—a classical RL concept—but at the level of natural language.
-
RL Update (Fine-tuning): Once successful prompts are discovered, the policy is fine-tuned using Proximal Policy Optimization (PPO). To ensure the model still works with the original "canonical" instruction, the authors use Mixed Backpropagation. This technique couples the exploratory prompt and the original prompt during training, forcing the model to ground the successful behavior back to the original task description.
Datasets
The researchers evaluated PDE across a diverse set of benchmarks:
- Robotics (Simulation): LIBERO and LIBERO-PRO, which focus on lifelong learning and generalization in manipulation, and ManiSkill, a tabletop pick-and-place benchmark.
- Robotics (Real-world): A physical Franka FR3 arm performing 10 different language-conditioned manipulation tasks.
- Reasoning/Coding: LiveCodeBench (competitive programming) and AIME 2026 (mathematical reasoning) to demonstrate that PDE works beyond just robotics.
Key Insights and Ideas
- Prompts as Global Perturbations: Unlike action noise, which is local and transient, changing a prompt induces a "global" change in behavior that persists throughout the entire rollout. It changes the intent of the agent, not just its reflexes.
- Language as a Tractable Interface: Performing Bayesian inference over the billions of parameters in a foundation model is mathematically intractable. However, performing inference over the space of prompts using a pre-trained VLM is efficient and leverages the VLM's existing world knowledge.
- Bootstrapping from Zero: The most striking insight is that PDE can solve tasks where standard RL fails completely. By finding a "paraphrase" that the model happens to understand better, PDE generates the first successful examples needed to start the RL engine.
- Semantic Robustness: Training on a diverse "pool" of exploratory prompts actually makes the final policy more robust to different ways of phrasing the same command (paraphrase robustness).
Key Results
- Escaping Zero Success: On "Hard" tasks in LIBERO-PRO where the initial policy had 0% success, PDE bootstrapped to nearly 100% success, while standard action-noise methods remained stuck at zero.
- Sample Efficiency: In ManiSkill, PDE outperformed the best action-noise baselines across all generalization categories (visual, semantic, and execution).
- Real-World Success: On a physical Franka robot, PDE improved average task success from 20% to 55% in just 3.5 hours of on-robot training, more than doubling the performance of the action-noise baseline (25%).
- Beyond Robotics: On LiveCodeBench, the PDE-enhanced agent reached 50% accuracy twice as fast as standard RL methods, showing that "rethinking the problem description" is a universal exploration strategy.
Limitations and Future Work
- VLM Dependency: PDE relies on a "capable" VLM supervisor. If the supervisor cannot accurately diagnose why a rollout failed or suggest a meaningful alternative, the exploration loop breaks.
- Computational Overhead: The prompt optimization stage adds an extra loop of inference and rollout before the actual RL training begins.
- The "Regime 3" Problem: The authors identify a regime where even prompt optimization fails—if the policy simply lacks the motor skills to perform a task regardless of the instruction, PDE cannot help. Future work might involve combining prompt exploration with more traditional skill-learning approaches.
Summary updated