Abstract
Embodied Large Language Models (LLMs) endow robots with high-level task reasoning, but they typically lack the ability to reflect on failures, leading to repeated mistakes in deployment. We introduce Reflective Test-Time Planning (RTTP), a framework that integrates two modes of reflection: reflection-in-action, where the agent uses test-time scaling to simulate and score candidate actions; and reflection-on-action, which uses test-time training to update the model based on external feedback. By incorporating retrospective reflection for long-horizon credit assignment, RTTP allows agents to learn from errors in real-time. Experiments on our Long-Horizon Household and MuJoCo benchmarks show significant gains, with success rates nearly tripling over state-of-the-art baselines.
Introduction
While LLMs have significantly advanced robotic reasoning, they often function as "static oracles." When a robot fails at a task—such as attempting to fit an object into an overfilled container—it frequently repeats the same error in subsequent trials because it cannot reflect on the underlying cause of failure.
This paper identifies two primary gaps in current embodied LLMs: the lack of internal deliberation (greedy action selection) and the "credit assignment problem," where a failure at step 20 may be caused by a strategically poor choice at step 2. RTTP addresses these by enabling robots to "think before doing" and "learn by doing."
Reflective Test-Time Planning
The RTTP framework consists of an Action LLM, an Internal Evaluator, and an External Evaluator. It operates through a tripartite reflective process:
- Reflection-in-Action (Test-Time Scaling): Before execution, the robot samples $N$ candidate actions. The internal evaluator "mentally simulates" these actions, scoring them to select the most promising path.
- Reflection-on-Action (Test-Time Training): After an action is executed, an external evaluator provides a natural language critique (e.g., "The box is too small"). The robot uses Low-Rank Adaptation (LoRA) to update its weights during the task, learning immediately from the feedback.
- Retrospective Reflection (Hindsight): Periodically, the robot re-evaluates its entire history. This allows it to realize that an earlier action, which appeared successful at the time, was actually the root cause of a later failure. This hindsight is used to refine both the policy and the internal evaluator.
Experimental Setup
We evaluated RTTP across several challenging environments:
- Long-Horizon Household Benchmark: A new benchmark based on BEHAVIOR-1K featuring over 1,000 household scenes testing fitting, selection, and sequential constraints.
- MuJoCo Cupboard Fitting: A controlled environment for measuring geometric placement precision.
- Habitat-Matterport 3D (HM3D): Testing generalization in photorealistic indoor environments.
- Real-Robot Trials: Validation on a physical Franka Panda robotic arm.
Key Results and Insights
The framework demonstrated substantial improvements over existing methods:
- Success Rates: RTTP achieved a 33.65% average success rate on the Household benchmark, nearly tripling the 11.2% achieved by the Self-Refine baseline.
- Geometric Reasoning: In "Fitting" tasks, performance jumped from 10.6% to 44.7%, highlighting the effectiveness of verbal critiques in teaching geometric constraints.
- Synergy of Scaling and Training: Ablation studies revealed that internal simulation and test-time training are mutually dependent; without training, the internal evaluator is often overconfident, and without scaling, the robot lacks sufficient exploration to learn effectively.
Conclusion and Future Work
RTTP proves that reflective planning is a powerful tool for embodied agents, allowing them to break the cycle of repetitive errors. While the current method increases per-step computation, the resulting "reflective time" is significantly more valuable than additional trial-and-error steps. Future research will focus on optimizing reflection triggers to reduce latency and integrating multi-modal feedback, such as tactile sensing, to detect non-visual failures.
Summary updated