TL;DR
RePlan-VLA is a framework that enables Vision-Language-Action (VLA) models to tackle complex, multi-step tasks by wrapping them in an execution-aware planning loop. By using a specialized progress-monitoring VLM to detect failures and trigger adaptive replanning, it significantly boosts success rates on long-horizon tasks without requiring any retraining of the underlying robot policy.
What Problem Does This Paper Address?
Modern Vision-Language-Action (VLA) models are impressive at "short-horizon" tasks—simple, direct instructions like "pick up the red block." However, they tend to fall apart when faced with "long-horizon" instructions that require a sequence of interdependent steps (e.g., "Open the drawer, get the straw, and put it in the cup").
The core issue is that these models are largely reactive; they lack the high-level reasoning, memory, and progress tracking needed to manage multi-stage goals. Small errors in the first few steps—like a slightly missed grasp—propagate and snowball, leading to total task failure. Most existing solutions try to "burden" the VLA with these complex reasoning capabilities through expensive fine-tuning, which often degrades the model's basic manipulation skills.
Methods
Instead of building a bigger, more complex VLA, the authors propose RePlan-VLA, a framework that decouples high-level reasoning from low-level execution. It treats the VLA as a "frozen" executor of short-horizon skills and manages the "big picture" through three main components:
- Planner VLM ($D_\Psi$): A few-shot Vision-Language Model (Qwen-VL-8B) that decomposes a long-horizon instruction into a sequence of executable sub-tasks. Crucially, it can revise this plan on the fly if things go wrong.
- Progress Estimator VLM ($F_\Theta$): This is the "eyes" of the system. It’s a VLM fine-tuned to monitor execution at the level of manipulation primitives (e.g.,
go_tofor approaching/grasping andmove_tofor transporting). It evaluates whether a sub-task is progressing or if a failure has occurred. - Frozen VLA Policy ($\pi_\Phi$): The low-level workhorse (e.g., Awomo or $\pi_{0.5}$) that translates the current sub-task instruction into actual robot motor commands.
The system also includes an Instruction Verifier that ensures the planner's sub-tasks are phrased in a language the VLA actually understands, preventing "lost in translation" errors between the high-level planner and the low-level controller.
Datasets
The authors evaluated RePlan-VLA across several challenging environments:
- RoboCasa365: A large-scale simulation benchmark for kitchen manipulation tasks, testing both "Seen" and "Unseen" task combinations.
- CALVIN: A standard benchmark for long-horizon, language-conditioned manipulation. The authors used a specifically difficult subset called "hardD" to push the limits of the system.
- Real-World Deployment: The framework was validated on a Franka Research 3 robotic arm performing 3-task sequences.
Key Insights and Ideas
- Primitive-Level Monitoring is Key: Most previous work tried to track overall task progress. RePlan-VLA monitors the "primitives" (the building blocks of a sub-task). This fine-grained view allows the system to catch errors—like grabbing a cupcake instead of a jar—much earlier and provide specific feedback for recovery.
- VLA-Agnostic Design: You don't need to touch the weights of the expensive robot policy. This modularity means RePlan-VLA can be "wrapped" around any existing or future VLA model to immediately improve its long-horizon performance.
- Language as a Diagnostic Tool: When the Progress Estimator detects a failure, it generates a natural language explanation (e.g., "The robot is currently holding the lemon instead of the onion"). This allows the Planner VLM to use its inherent reasoning capabilities to figure out exactly how to fix the situation.
Key Results
RePlan-VLA demonstrated significant performance gains over state-of-the-art baselines:
- RoboCasa365: Achieved a 62.5% average success rate, outperforming the next strongest baseline (Awomo-0.5I) by 21.9 percentage points.
- CALVIN: On the "hardD" sequence, it completed an average of 4.10 out of 5 consecutive tasks, compared to 3.58 for the best baseline.
- Generalization: On "Unseen" tasks in RoboCasa365, it maintained a 56.3% success rate, while the next best model dropped to 26.3%, highlighting the robustness of the replanning approach.
- Efficiency: The system adds manageable overhead, with an average end-to-end execution time of approximately 0.58 seconds per environment step.
Limitations and Future Work
- Irreversible Failures: The system can plan its way out of a missed grasp, but it cannot recover if the robot physically breaks an object or knocks it into an unreachable area.
- Visual Grounding Reliance: The planner's success depends heavily on the quality of object detection and scene annotations. Errors in the initial "scene observation" phase can lead to flawed plans.
- Future Directions: The authors suggest incorporating tactile sensing to improve progress estimation (e.g., "feeling" if an object is actually in the gripper) and exploring skill-aware replanning that accounts for the specific strengths and weaknesses of different VLA policies.
Summary updated