Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs

Yining Hong, Huang Huang, Manling Li, Li Fei-Fei, Jiajun Wu, Yejin Choi

Abstract

Embodied Large Language Models (LLMs) endow robots with high-level task reasoning, but they typically lack the ability to reflect on failures, leading to repeated mistakes in deployment. We introduce Reflective Test-Time Planning (RTTP), a framework that integrates two modes of reflection: reflection-in-action, where the agent uses test-time scaling to simulate and score candidate actions; and reflection-on-action, which uses test-time training to update the model based on external feedback. By incorporating retrospective reflection for long-horizon credit assignment, RTTP allows agents to learn from errors in real-time. Experiments on our Long-Horizon Household and MuJoCo benchmarks show significant gains, with success rates nearly tripling over state-of-the-art baselines.

Introduction

While LLMs have significantly advanced robotic reasoning, they often function as "static oracles." When a robot fails at a task—such as attempting to fit an object into an overfilled container—it frequently repeats the same error in subsequent trials because it cannot reflect on the underlying cause of failure.

A robot must reason about geometric constraints, such as fitting objects into containers already occupied by items like this teddy bear.

This paper identifies two primary gaps in current embodied LLMs: the lack of internal deliberation (greedy action selection) and the "credit assignment problem," where a failure at step 20 may be caused by a strategically poor choice at step 2. RTTP addresses these by enabling robots to "think before doing" and "learn by doing."

Reflective Test-Time Planning

The RTTP framework consists of an Action LLM, an Internal Evaluator, and an External Evaluator. It operates through a tripartite reflective process:

  1. Reflection-in-Action (Test-Time Scaling): Before execution, the robot samples $N$ candidate actions. The internal evaluator "mentally simulates" these actions, scoring them to select the most promising path.
  2. Reflection-on-Action (Test-Time Training): After an action is executed, an external evaluator provides a natural language critique (e.g., "The box is too small"). The robot uses Low-Rank Adaptation (LoRA) to update its weights during the task, learning immediately from the feedback.
  3. Retrospective Reflection (Hindsight): Periodically, the robot re-evaluates its entire history. This allows it to realize that an earlier action, which appeared successful at the time, was actually the root cause of a later failure. This hindsight is used to refine both the policy and the internal evaluator.

Experimental Setup

We evaluated RTTP across several challenging environments:

  • Long-Horizon Household Benchmark: A new benchmark based on BEHAVIOR-1K featuring over 1,000 household scenes testing fitting, selection, and sequential constraints.
  • MuJoCo Cupboard Fitting: A controlled environment for measuring geometric placement precision.
  • Habitat-Matterport 3D (HM3D): Testing generalization in photorealistic indoor environments.
  • Real-Robot Trials: Validation on a physical Franka Panda robotic arm.
The Long-Horizon Household benchmark provides complex, multi-room environments to test the limits of robotic reasoning and reflection.

Key Results and Insights

The framework demonstrated substantial improvements over existing methods:

  • Success Rates: RTTP achieved a 33.65% average success rate on the Household benchmark, nearly tripling the 11.2% achieved by the Self-Refine baseline.
  • Geometric Reasoning: In "Fitting" tasks, performance jumped from 10.6% to 44.7%, highlighting the effectiveness of verbal critiques in teaching geometric constraints.
  • Synergy of Scaling and Training: Ablation studies revealed that internal simulation and test-time training are mutually dependent; without training, the internal evaluator is often overconfident, and without scaling, the robot lacks sufficient exploration to learn effectively.

Conclusion and Future Work

RTTP proves that reflective planning is a powerful tool for embodied agents, allowing them to break the cycle of repetitive errors. While the current method increases per-step computation, the resulting "reflective time" is significantly more valuable than additional trial-and-error steps. Future research will focus on optimizing reflection triggers to reduce latency and integrating multi-modal feedback, such as tactile sensing, to detect non-visual failures.

Summary updated

Abstract

Embodied LLMs endow robots with high-level task reasoning, but they cannot reflect on what went wrong or why, turning deployment into a sequence of independent trials where mistakes repeat rather than accumulate into experience. Drawing upon human reflective practitioners, we introduce Reflective Test-Time Planning, which integrates two modes of reflection: \textit{reflection-in-action}, where the agent uses test-time scaling to generate and score multiple candidate actions using internal reflections before execution; and \textit{reflection-on-action}, which uses test-time training to update both its internal reflection model and its action policy based on external reflections after execution. We also include retrospective reflection, allowing the agent to re-evaluate earlier decisions and perform model updates with hindsight for proper long-horizon credit assignment. Experiments on our newly-designed Long-Horizon Household benchmark and MuJoCo Cupboard Fitting benchmark show significant gains over baseline models, with ablative studies validating the complementary roles of reflection-in-action and reflection-on-action. Qualitative analyses, including real-robot trials, highlight behavioral correction through reflection.
BibTeX
@misc{hong2026learning,
  author = {Yining Hong and Huang Huang and Manling Li and Li Fei-Fei and Jiajun Wu and Yejin Choi},
  title = {{Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs}},
  year = {2026},
  eprint = {2602.21198},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.21198},
}
Figure