Language-model planners for household robots often write plans that read well but fail on contact with the world. They grasp an item inside a closed drawer, or pick something up with their hands already full. A diagnostic benchmark attributes 45.2% of such failures to trajectory feasibility (Embodied Agent Interface). The usual explanation is reporting bias: text rarely states the obvious physical steps, so the model never learned them. This proposal tests a different explanation and builds on it. Many of these errors may be failures to bring up knowledge the model already holds in its weights, not missing knowledge. If so, a small nudge to the model's internal activations should fix them, and the right nudge for a given physical situation can be looked up in a growing external memory built automatically from past execution failures.
Concretely, every time a simulated plan fails because a physical precondition was violated, we record the situation and the corrected action. We convert the gap between the wrong and right continuation into an activation steering vector: a direction added to the model's hidden states during the forward pass that shifts what it generates. That vector is stored under a key describing the physical state. At test time, in new environments and with no feasibility checker available, the planner retrieves the vectors whose stored states match its current one and is steered by them. Before any of that, the project measures which failures are recoverable by steering at all and which need actual world-state information. That measurement is what makes the method's success or failure informative.
Core Claim
We claim that a substantial and measurable share of embodied feasibility errors are elicitation failures. In those cases the correct, precondition-respecting action is already near the top of the model's ranked predictions, or is linearly decodable from its mid-layer states, but it loses to a more "textually typical" action. For this subset, retrieving state-keyed steering vectors from a failure memory should cut errors without retraining. It should also transfer across environments better than fine-tuning and scale with memory size better than putting the same failures into the prompt. For the complementary subset, where the model lacks environment-specific facts, steering should not help, and the gains should be concentrated accordingly. The method claim is useful only if the mechanism claim holds, and the experiments test both.
Position Relative to Prior Work
The method is a direct descendant of Dynamic Steering With Episodic Memory (DSEM). DSEM stores steering vectors computed from human-written demonstration pairs in a key-value memory, retrieves them by text similarity, and blends a retrieved vector with a global one. It reduces ParaDetox toxicity on Gemma-9b to 4.29% from 44.28%, and it improves monotonically as memory grows from 10 to 100 entries. Our proposal differs in three ways. First, DSEM's keys are conversational text and its domain is style and toxicity, while ours are physical states. Second, its entries are curated demonstrations, while ours are mined automatically from execution failures, so the memory grows with experience. Third, we push memory size two orders of magnitude past DSEM's tested range.
Dynamic Multimodal Activation Steering is the other close relative. It retrieves topic-clustered truthfulness vectors to reduce hallucination in vision-language models, but it addresses static image questions with a fixed cache, not sequential action under changing state.
Two lines of work supply the diagnostic idea. ARGS reweights tokens within a top-10 candidate set using a reward model, and its success indicates that the desired behavior often sits among near-miss predictions. Are LLMs Really Not Knowledgeable? and Dynamic Focus Decoding argue that much apparent ignorance is a retrieval failure inside the model. Dynamic Focus Decoding detects this through layer-to-final-layer divergence and uses it only to set decoding temperature. We borrow these signals as measurement tools for physical knowledge, where they have not been applied.
A constraining result comes from Generated Knowledge Prompting. There, 17% of self-generated facts were wrong, and those wrong facts caused 86% of the resulting failures. This is why we do not ask the planner to write out its own physical commonsense. The corrective signal must come from real execution outcomes, not from the model's own claims.
Three embodied systems motivate the setting and serve as contrasts. SayCan multiplies a language score by a learned feasibility score over a fixed library of 551 skills, reaching 84% planning success. It leaves the language model's internals untouched and needs a feasibility function at test time. ExRAP and Planning with an Embodied Learnable Memory keep external world-state memories, a time-decayed knowledge graph and a text object-state log respectively, and pass them through the prompt. Their memories store facts about the world, whereas ours stores corrections to the model's behavior. The two are complementary, and the second is precisely what our diagnostic predicts is needed for the non-elicitation failures.
Research Program
The work has three stages. Each one gates the next.
Stage one: diagnose failures. We take an open-weight planner, Llama-3.1-8B (the model family used by Dynamic Focus Decoding), and collect its failures on the action-sequencing and transition-modeling modules of the Embodied Agent Interface over VirtualHome and BEHAVIOR. That benchmark already labels errors as missing steps, additional steps, and feasibility violations. For each failure we ask three questions. Is the simulator-validated correct action within the model's top-10 candidates? Can a linear probe trained on held-out environments read "unmet precondition" off a mid-layer state? Does layer divergence on the action tokens look like confident recall or like guessing? A failure that passes the first two tests is labeled an elicitation failure. The output of this stage is a measured split of errors into elicitation and ignorance failures, broken down by error category. If elicitation failures are rare, the project pivots to the world-state memory route and does not proceed with steering.
Stage two: build the failure memory. We run the planner in training environments with the simulator in the loop. When an action fails a precondition, we pair the failing step with the corrected action. The corrected action is either the next step of the expert plan or the action that resolves the simulator's error, for example "open drawer". We compute the steering vector as the difference in residual-stream activations at a chosen layer between the prompt continued with the corrected action and the prompt continued with the failing one. The key is the planner's own final-token hidden state for that situation, augmented with a short symbolic signature: held object, relevant container states, and pending subgoal. We store keys and vectors in a nearest-neighbor index of the kind used by kNN-LM.
At inference, we retrieve the top-$k$ entries. We weight them by key similarity and blend them with a global average vector, in the manner of DSEM. The combined vector is added at the chosen layer with a scale $\alpha$ tuned on validation environments. Steering applies only to tokens inside the action slot, so the planner's free-text reasoning is not perturbed.
Stage three: scale and transfer. We grow the memory from 10 to about 10,000 entries and measure success, retrieval latency, and degradation of general capability. We then test the memory in environments and datasets it was not built from.
Baselines
The naive reference is the same frozen planner with standard prompting and no memory. Every gain is reported relative to it, broken down by the error categories from stage one.
The strongest prompt-based competitor is text retrieval of the same failure records. The retrieved (situation, wrong action, corrected action) triples are written into the context, as in standard retrieval-augmented prompting. This baseline uses the identical memory and the identical retriever and retrieval depth, so the only difference from our method is whether a correction enters as text or as an activation offset. Our method beats it only if it achieves higher task success at equal memory size and keeps improving at memory sizes where the prompted version plateaus or degrades.
The strongest weight-update competitor is LoRA fine-tuning of the same base model on the corrected trajectories. It gets the same failure data and the same hyperparameter-search budget as our scale and layer search. LoTa-Bench gives the reference behavior: fine-tuning LLaMA-1 30B on ALFRED raised in-domain success from 13.7% to over 60%, but the model dropped by over 20 percentage points when evaluated on WAH-NL. Beating this baseline means matching it within a few points in-domain while losing substantially less on the cross-dataset transfer.
For absolute context on these tasks, LoTa-Bench reports 40.38% ALFRED task success for GPT-4 and 43.25% subgoal success on WAH-NL for LLaMA-1 65B with semantic-similarity example selection. The Embodied Agent Interface reports 74.9% average BEHAVIOR performance for o1-preview. These are reference points, not comparisons at matched model size. No reported result was found on Lacuna for any activation-steering method on an embodied planning benchmark, so the steering baselines below are run by us.
Two ablations isolate the key ingredients. A single global steering vector, the mean of all memory entries applied everywhere in the style of contrastive activation addition, tests whether state-keyed retrieval matters at all. A random-key control retrieves vectors from mismatched states, testing whether gains come from situation-specific content rather than generic "be careful" directions.
Finally, an oracle masking baseline zeroes out actions the simulator declares infeasible at test time, in the spirit of SayCan's gating. It serves as an upper bound on single-step feasibility under privileged information that our test regime deliberately withholds.
Why not just…?
Why not just retrieve the past failures as text and put them in the prompt? This is the most serious threat, since it needs no access to the model's internals and works with closed models. The file gives indirect reasons to expect it to saturate. DSEM reports that in-context demonstrations plateau as their number grows while memory-based steering keeps improving. Retrieval-Augmented Robots shows GPT-4o assembly F1 falling from 0.736 to 0.186 as manuals grow from 1–5 to 16+ parts. Neither result is about embodied failure memories, though, so the matched comparison in our experiments settles the question. If text retrieval equals steering at every memory size, we will report that the activation pathway adds nothing here.
Why not just fine-tune on the corrected trajectories? In-domain, fine-tuning may well win, and we expect it to. The case against it is transfer and flexibility. LoTa-Bench's drop of over 20 points on unseen data suggests fine-tuning memorizes dataset-specific shortcuts. A memory can also be edited or extended per deployment without retraining. The ALFRED-to-WAH-NL experiment tests this directly.
Why not just mask infeasible actions with a checker? When a reliable feasibility checker exists at test time, masking is simpler and probably better at single-step validity, and we report it as an upper bound for that reason. Our claim applies to the regime where no checker is available, which is the case on a real robot or whenever preconditions surface only through failure. Masking also cannot make the planner propose a missing precondition step that is absent from its candidate list. Comparing our method's effect on the missing-step category against oracle masking will show whether that distinction matters in practice.
Why not just use a confidence threshold to decide when to look things up? If stage one finds that low output probability predicts elicitation failures as well as the probe and layer-divergence signals do, we will use the simpler signal. The diagnostic is there to determine which mechanism is operating, not to promote a particular signal.
Proposed Experiments and Decisive Checks
The primary benchmarks are the Embodied Agent Interface modules on VirtualHome and BEHAVIOR, measured by task success, executability, and per-category error rates. They are complemented by LoTa-Bench ALFRED and WAH-NL, measured by task and subgoal success. Memory is always built from training environments only and evaluated in unseen ones. We also check capability retention on TruthfulQA and a held-out instruction-following set, since coarse steering is known to damage fluency (Fine-Grained Activation Steering).
The first decisive check is mechanistic. Error reductions should fall overwhelmingly on failures that stage one labeled as elicitation failures, with near-zero effect on ignorance failures. A uniform improvement would undercut the explanation even if success rose.
The second check is the matched text-versus-vector comparison across memory sizes from 10 to 10,000. The method earns its complexity only if the steering curve stays above or continues past the prompting curve.
The third check is transfer: building on ALFRED and testing on WAH-NL must lose clearly less than the fine-tuned baseline.
The fourth check is the random-key ablation, which must perform substantially worse than true retrieval.
If all four checks hold, the result is a practical tool and evidence for a specific claim: part of the physical grounding gap is an elicitation problem, and it can be corrected at inference time by an experience-built memory.
References
- Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
- Dynamic Steering With Episodic Memory For Large Language Models
- Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
- ARGS: Alignment as Reward-Guided Search
- Are LLMs Really Not Knowledgeable? Mining the Submerged Knowledge in LLMs' Memory
- Odysseus Navigates the Sirens' Song: Dynamic Focus Decoding for Factual and Diverse Open-Ended Text Generation
- Generated Knowledge Prompting for Commonsense Reasoning
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
- Exploratory Retrieval-Augmented Planning For Continual Embodied Instruction Following
- Planning with an Embodied Learnable Memory
- Generalization through Memorization: Nearest Neighbor Language Models
- LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents
- Retrieval-Augmented Robots via Retrieve-Reason-Act
- Fine-Grained Activation Steering: Steering Less, Achieving More