Language-model planners for household robots are increasingly fine-tuned with reinforcement learning (RL): the model proposes a sequence of actions, a simulator scores it, and the model is nudged toward higher-scoring plans. This works well when nothing goes wrong. Real execution, however, is full of small failures. A grasp slips, an object is not where it was expected, a door is still closed. Recovering from these requires the model to do something different from what it just tried. We argue that the standard RL recipe quietly removes this ability. RL is known to make a model's output distribution very peaked, a failure mode called entropy collapse, meaning the model ends up putting nearly all its probability on one answer. In an embodied planner, a collapsed model that is told "that action failed" and asked again will propose the same failed action, because it no longer assigns meaningful probability to any alternative.
The proposal is to measure this directly and fix it where it matters. We will build a closed-loop version of an existing embodied RL benchmark with injected execution failures. We will show that collapse appears as an inability to recover rather than as lower nominal success. We will then train with two changes. First, groups of RL rollouts are anchored at failure states, so the learning signal is about choosing a recovery. Second, an entropy floor is applied only to decisions made right after a failure, which leaves the precise, low-entropy behavior on ordinary steps untouched. The claim is specific: recovery competence depends on the probability the policy keeps on feasible alternatives conditioned on an observed failure, and global entropy tricks protect the wrong quantity.
Where this sits
Four strands of prior work motivate the proposal, and one supplies its starting point. Reinforced Reasoning for Embodied Planning applies group-relative policy optimization (GRPO, which scores each rollout against the mean of a group of rollouts for the same prompt) to Qwen2.5-VL-7B after distilled supervised fine-tuning (SFT). It reaches 35.6% success on EB-ALFRED. It generates plans open-loop, as one JSON sequence, and never measures entropy or rollout diversity. This is our starting point and our main baseline. SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR shows on code generation that longer SFT lowers entropy and leaves GRPO groups homogeneous, so gradients vanish. Its 1.0-epoch checkpoint reaches pass@10 of 0.806 after RL, against 0.481 for the 5.8-epoch checkpoint. It also finds that KL penalties and label smoothing do not rescue collapsed checkpoints. That negative result constrains our design, since we cannot expect a global KL term to work. Embodied pipelines routinely run heavy behavior cloning before RL, but nobody has checked whether the same inversion happens there.
The entropy-control methods built for text reasoning are our nearest methodological relatives. AutoTool uses separate entropy constraints for short and long tool-use paths. CDE adds curiosity bonuses from perplexity and critic disagreement. SvS synthesizes problem variants to keep entropy up. All three condition their intervention on properties of the text, such as path length, surprise, or problem difficulty. None conditions on feedback from a physical environment, and all are evaluated only on math, code, or API calls. Our entropy floor is conditioned on whether the environment just reported a failure, which is information these methods do not have.
On the grounding and closed-loop side, CAPE re-prompts a zero-shot LLM with natural-language precondition failures and raises VirtualHome plan correctness from 28.89% (SayCan) to 49.63%. E²CL fine-tunes on deliberately collected error trajectories. Its 77M model beats a 783M model trained only on expert data, and it reports that plain PPO suffered from sparse rewards without asking why. Neurosymbolic Embodied Agents masks infeasible actions with a hand-written planning-domain model and reaches 94.5–99.5% on VirtualHome. It depends on a correct symbolic world model, and perception errors leave it with no recovery path. Open-loop VLM Robot Planning finds that self-verification prompts cut performance by over 10% on fine-tuned planners. That is consistent with our hypothesis that fine-tuned models cannot generate alternatives on request, though that study never closed the loop. Our proposal differs from all of these by studying RL-trained planners in a closed loop and asking whether training itself destroys the alternatives that re-prompting and error correction assume exist.
Core Claim
In RL-trained embodied planners, entropy collapse appears mainly as a high retry-repeat rate: the fraction of post-failure re-queries in which the model proposes the action that just failed. This happens even when nominal success is unchanged or rising. The cause is that states following a failure are rare in rollouts. As a result, GRPO groups starting from the task prompt almost never contain contrasting recovery decisions, and the policy's mass at those states concentrates on its single habitual action.
The fix has two parts. The first is failure-anchored groups. When a rollout hits a failure at state $s_f$, we resample $G$ continuations from $s_f$ and compute group-relative advantages among them. The second is an entropy floor applied only at those states:
$$ \mathcal{L} = \mathcal{L}_{\text{GRPO}} + \lambda \, \mathbb{E}_{s_f}\big[\max\big(0,\; H_0 - H(\pi(\cdot \mid s_f))\big)\big]. $$
Here $\mathcal{L}_{\text{GRPO}}$ is the standard group-relative loss and $s_f$ ranges over states immediately after an environment-reported failure. $H(\pi(\cdot \mid s_f))$ is the policy's entropy over the action-identity tokens of its next step, excluding JSON formatting tokens. $H_0$ is a target floor and $\lambda$ a weight.
The hinge leaves confident behavior alone when entropy is already high enough. Restricting the term to action-identity tokens at $s_f$ avoids the format breakage reported for sampling-based methods by Open-loop VLM Robot Planning.
Research Program
Diagnostic study. We first characterize the phenomenon without any new method. Starting from Qwen2.5-VL-7B, we reproduce the SFT-then-GRPO recipe of Reinforced Reasoning for Embodied Planning on EB-ALFRED. We save SFT checkpoints at several epoch counts and run identical GRPO on each.
For every checkpoint, before and after RL, we record five quantities:
- action-token entropy;
- the fraction of GRPO groups with zero advantage variance;
- pass@1 and pass@K on nominal tasks;
- the retry-repeat rate;
- recovery success under injected failures.
This tests two things at once. One is whether the rank inversion seen on code transfers to embodied planning, meaning that the best SFT pass@1 does not give the best post-RL agent. The other is whether retry-repeat rate rises over GRPO training while nominal success holds.
Closed-loop environment. We wrap the EB-ALFRED and VirtualHome simulators so that the planner emits one step at a time and receives a short textual outcome. With probability $p$, a step that would succeed is made to fail in a physically plausible way. Examples include a grasp that leaves the object in place, a target object moved to another receptacle, or a container that is re-closed. We use separate perturbation types for training and test, so we can check that recovery transfers to unseen failure types rather than to memorized fixes.
Method. We then train the failure-anchored objective from the same SFT checkpoints, holding the total number of environment steps fixed, and compare it against the alternatives below. The analysis question is which of three ingredients accounts for any recovery gain: anchoring the groups at failure states, the conditional entropy floor, or simply exposing the model to failures during training.
Baselines
The naive reference points are the distilled SFT model and the reproduced open-loop GRPO model, both run step-wise in our closed-loop wrapper. Reinforced Reasoning for Embodied Planning reports EB-ALFRED success of 21% for SFT and 35.6% for GRPO, against 22.0% for GPT-4o-mini and 32.0% for LLaMA-3.2-90B. We first confirm that our reproduction lands near 35.6% under nominal conditions. Under injected failures there is no reported result found on Lacuna for any method, so those numbers will be established here.
The strongest competitor on the specific question is GRPO trained with the same failure injection but standard prompt-anchored groups and no entropy term. It isolates the obvious explanation, that recovery simply needs failure exposure. It gets the identical environment-step budget, the same SFT checkpoints, and the same hyperparameter search budget.
We also run three entropy-control alternatives under the same injection and budget:
- GRPO plus a global entropy bonus;
- GRPO plus a KL penalty to the SFT model;
- an AutoTool-style adaptive constraint (AutoTool), with its long-path trigger replaced by trajectory length in steps.
These test whether any entropy protection suffices. The CDE curiosity bonus (CDE) is a further text-side comparison.
Two inference-time and supervised baselines come from the grounding literature. CAPE-style corrective re-prompting (CAPE) is applied on top of the GRPO model; CAPE itself reports 49.63% correctness and 96.29% executability on VirtualHome with a zero-shot LLM. An E²CL-style SFT on collected error trajectories (E²CL) is matched in the number of failure-containing trajectories. The PDDL-masked agent of Neurosymbolic Embodied Agents is reported as a privileged upper reference on VirtualHome only, since it uses a hand-built world model the other methods lack.
Two ablations isolate our ingredients: anchored groups without the entropy floor, and the floor without anchoring.
Beating the strongest competitor means two things. Recovery success on held-out perturbation types must be higher, with a non-overlapping confidence interval across three seeds. Nominal EB-ALFRED success must be no worse than 35.6% minus seed noise.
Why not just…?
Why not just inject failures during RL and let standard GRPO learn recovery? This is the most serious threat, and it is a named baseline. Our argument that it falls short is mechanistic. SFT Overtraining Predicts Rank Inversion shows that homogeneous groups give zero gradient. Prompt-anchored groups rarely share the same failure state, so recovery decisions are almost never contrasted against each other. We will measure the fraction of zero-variance comparisons at post-failure states under both schemes. If plain injection matches our method on held-out recovery, we will report that the entropy machinery is unnecessary.
Why not just add closed-loop re-prompting at inference, as CAPE does? CAPE works with an uncollapsed zero-shot model. Our hypothesis is that re-prompting an RL-collapsed model mostly re-issues the failed action. This is consistent with self-verification hurting fine-tuned planners in Open-loop VLM Robot Planning. The retry-repeat rate of the GRPO model under re-prompting tests this directly. If re-prompting alone restores recovery, the training-side claim is wrong.
Why not just stop SFT early and pick the highest-entropy checkpoint? The rank-inversion results suggest this alone may recover much of the post-RL performance, so it is a real threat. The SFT-epoch sweep makes it an explicit arm. The open question is whether an early-stopped checkpoint keeps its entropy at failure states through long GRPO training, or collapses there anyway. If early stopping plus plain GRPO matches us, that becomes the headline finding.
Why not just use a global entropy bonus or KL penalty? KL penalties already failed to rescue collapsed checkpoints in SFT Overtraining Predicts Rank Inversion. A global bonus spends exploration on formatting and routine steps, where precision matters. The global-bonus arm tests whether it raises recovery only at the cost of nominal success and output validity.
Proposed Experiments and Decisive Checks
The first decisive check is the diagnostic. If retry-repeat rate does not rise during standard GRPO on EB-ALFRED, and recovery under injection does not fall behind nominal success, then the phenomenon we target is absent and the method is unmotivated. We would publish the negative result and stop there.
The second check is the rank-inversion sweep. A negative or zero correlation between SFT pass@1 and post-RL recovery success, alongside a positive correlation for pre-RL action entropy, would confirm that the code-domain effect carries over to embodied planners.
The third check is the head-to-head against failure-injected GRPO and the two ablations, measured on held-out perturbation types. To test that we are not overfitting to one simulator, we evaluate on VirtualHome after training on EB-ALFRED, and on EB-Habitat for unseen environments. Throughout, we report nominal success, recovery success, retry-repeat rate, pass@K, the fraction of zero-variance groups, and JSON validity. For the analytical progress metrics, we use AgentBoard.
References
- Reinforced Reasoning for Embodied Planning
- SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR
- AutoTool: Automatic Scaling of Tool-Use Capabilities in RL via Decoupled Entropy Constraints
- CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
- Beyond Pass@ 1: Self-Play with Variational Problem Synthesis Sustains RLVR
- Planning With Large Language Models Via Corrective Re-Prompting
- E²CL: Exploration-based Error Correction Learning for Embodied Agents
- Neurosymbolic Embodied Agents
- Open-loop VLM Robot Planning: An Investigation of Fine-tuning and Prompt Engineering Strategies