Household robots driven by large language models can usually recall a single fact about their user ("she drinks from the blue mug"). They break down when one plan has to combine several such facts ("set up her light breakfast the way she likes, at her usual seat"). On the MEMENTO benchmark, GPT-4o loses 30.5% of its success rate when moving from single-memory to joint-memory tasks, and Llama-3.1-8B loses nearly 60% (Embodied Agents Meet Personalization). The same paper finds that retrieving more past interactions makes the problem worse, and that failing agents often fall back on generic "commonsense" choices, such as any cup instead of the user's cup. This resembles a failure seen far from robotics. In TheoremQA, putting the needed theorem directly into the prompt did not help models apply it. Knowledge that is present in the text is not necessarily used.
We propose to stop delivering personal memories as text. Instead, each memory becomes a compact internal signal injected at the exact decision where it matters. A frozen language model already compresses a few demonstrations into a "task vector": a hidden state at a middle layer that, when copied into a fresh forward pass, makes the model perform the demonstrated task without seeing the demonstrations (In-Context Learning Creates Task Vectors). Our idea is to extract one such vector per remembered user fact. The joint task is then split into steps, and at each step only the relevant fact's vector is patched in, at the token where the robot names an object or location. The two halves fit each other. Task vectors have only been shown to work for single-token outputs. Splitting the plan into steps reduces each personal decision to roughly a single-token choice, which is the regime where task vectors are known to work.
Core Claim
Joint-memory failure in personalized embodied planning has three parts: personal facts interfere with each other in a shared context, the agent applies them weakly against its commonsense prior, and the context grows too long. Step-scoped activation patching addresses all three. It keeps unrelated memories out of each decision, and it delivers each memory through a channel known to override textual context. In conflict experiments, a patched task vector beat contradictory in-context demonstrations (In-Context Learning Creates Task Vectors). The concrete, falsifiable prediction concerns an open-weights 8B model on MEMENTO. With step-scoped preference vectors, the gap between single-memory and joint-memory success should shrink substantially compared with the strongest text-based method under the same retrieval. The share of failures caused by commonsense override should also fall.
Relation to Prior Work
Three lines of work motivate the problem. MEMENTO measures the joint-memory collapse, the harm from retrieving more memories, and the commonsense trap. Its fix is a hierarchical knowledge graph of user facts that is still serialized into text. It also reports that aggressive summarization hurts small models because raw episodes carry useful action templates. This is a warning we take seriously: a vector must keep the template, not just the gist. In-situ Value-aligned HRI with Physical Constraints keeps a textual preference profile that is periodically condensed by "introspection." Even with iterative feedback, its logical feasibility tops out at 26.45%, which suggests the language-level planner, not the geometry solver, is the bottleneck. PARSEC turns past arrangements into JSON rules and frames placement as code completion. It reaches 0.59 placement accuracy and degrades in crowded scenes. All three keep preferences in the prompt; we move them into activations.
Two lines of work supply the method. Task vectors give us the extraction and patching procedure, including two facts we rely on: the "sweet spot" middle layer, and the finding that a vector built with a dummy query still works on a different query. Divide-and-conquer prompting (When Splitting Makes Stronger) shows that solving subproblems in isolated contexts reduces drift and compounding error. It also warns that the guarantees assume nearly disjoint subproblems. Joint-memory tasks are coupled, since the dish must go to the seat, so we keep a shared plan skeleton in text and isolate only the personal-fact decisions.
Two results constrain the design. The closest prior use of a learned preference vector is Transformers Are Adaptable Task Planners. It trains a dedicated encoder that turns one full-state demonstration into a preference embedding. That embedding reaches 0.62 spatial preference adherence on dishwasher loading and 0.54 on unseen preferences. We instead read vectors out of a frozen general model, from logged episodes, and compose several of them. Suppression Sticks, Locality Is Fragile shows that weight-space task-vector edits to robot policies damage unrelated skills, which kept only 52% of their success on average. Its authors explicitly leave activation-space steering untested. That result is why we patch activations transiently, step by step, and never edit weights or sum vectors globally. Variational Task Vector Composition finds that naive composition causes interference. This motivates our ablation that sums all vectors at once.
Research Program
The planner is Llama-3.1-8B, frozen. We chose it because it is the model that collapses most on MEMENTO and because patching needs access to its weights. A text-only plan skeleton is first produced by decomposition: the model lists steps and marks each slot that depends on a personal fact, together with the memory key it needs.
For each memory gathered in MEMENTO's acquisition stage, we build an extraction prompt. It contains the raw acquisition episode, so action templates are kept, and ends in a dummy query. We record the hidden state of the final token at a middle layer, choosing the layer by sweeping on a held-out split. When the planner fills a marked slot, the matching vector replaces the residual state at that layer for the slot's first argument token, and nothing else in the context changes.
Memory keys are resolved two ways. The first uses the same retriever every text baseline gets. The second uses oracle keys, so that retrieval error can be separated from the effect of delivering memories through activations.
The analysis has three parts. The first records whether each failure is a commonsense override (a generic object chosen), the wrong user fact, or a planning error. The second is a locality audit modelled on the VLA study: we patch a user's vector and check that slots for unrelated objects and non-personal steps are unchanged. The third is a logit-lens reading of the vectors, which projects intermediate hidden states onto the vocabulary to see whether they point to the user's specific object or receptacle names.
A secondary transfer study applies the same extraction to PARSEC. The vector comes from a user's prior arrangement, and placements for unplaced items are completed by patching. This tests whether the mechanism covers scene-derived preferences as well as dialogue memories.
Baselines
The naive reference is the same 8B planner with no memory beyond the current instruction. This shows how much success the generic commonsense prior alone achieves, and we expect it to be high on single-memory episodes that commonsense happens to satisfy.
The second reference is raw top-k text retrieval, swept over k as MEMENTO did, with the best k chosen on a validation split. It reproduces the published finding that more retrieval hurts.
The strongest existing method is MEMENTO's hierarchical knowledge-graph user-profile memory, serialized into the prompt. It is that paper's proposed solution to exactly this failure. The file reports relative drops (30.5% for GPT-4o and about 60% for Llama-3.1-8B on joint-memory tasks) but no absolute success rate for the knowledge-graph memory on joint-memory episodes. We found no reported result on Lacuna for that configuration, so we will re-run it ourselves with the same 8B model, retriever, episodes and prompt-tuning budget.
The most dangerous competitor is decomposition with text. It uses the identical plan skeleton and slot marking, but each slot gets its retrieved memory as text in an isolated context, in the style of divide-and-conquer prompting. It differs from our method only in how the memory is delivered.
Three ablations isolate the key ingredient. In the first, all retrieved vectors are summed and patched at every step, which removes step scoping. In the second, vectors come from a different user's memory of the same type. In the third, the vector is random but norm-matched.
On PARSEC the strongest reported baseline is ContextSortLM, at 0.59 placement accuracy (PARSEC). We will re-run it with our 8B backbone so that the backbone is held constant. On MEMENTO, beating the comparison means two things relative to decomposition with text under matched retrieval: a statistically significant reduction in the single-to-joint gap, and a lower commonsense-override rate, with single-memory success within noise. On PARSEC, beating it means matching or exceeding the re-run ContextSortLM with a fraction of its context tokens.
Why not just…?
Why not just decompose the task and give each step its memory as text? This is the real threat, and it could plausibly close most of the gap by itself, because the decomposition result shows that isolated contexts already reduce drift. Our bet rests on the commonsense trap, which is a failure to apply knowledge rather than a failure of context noise. TheoremQA's finding that supplying the rule in text does not help, together with the task-vector finding that patched vectors override textual context, suggests that delivery through activations is stronger. The head-to-head comparison, run with oracle keys, settles this. If decomposition with text matches us on both success and override rate, we will report that the benefit comes from the decomposition alone, and that vectors only save tokens.
Why not just use a better structured memory, such as MEMENTO's knowledge graph or ICLHF-style introspection summaries? Both still end up as prompt text that competes with the model's prior. MEMENTO itself reports that compressing raw episodes harms small models. The knowledge-graph baseline is run directly, and the failure taxonomy will show whether its remaining errors are commonsense overrides, which is the failure our method targets.
Why not just fine-tune a LoRA adapter per user? Per-user data is tiny and keeps changing, and weight-space personalization has known locality problems in robot policies (Suppression Sticks). The federated study also found that small fine-tuning deltas hide client-specific signal (FedBip). Even so, we include a per-user LoRA trained on the same acquisition episodes with a matched tuning budget, and evaluate it on joint success and on the locality audit. If it wins on both, our argument fails.
Why not just use a bigger model with a long context? GPT-4o still drops 30.5% on joint-memory tasks, so scale alone does not remove the gap. We will report GPT-4o under text decomposition as a reference ceiling, not as a fair competitor.
Proposed Experiments and Decisive Checks
The first check is a gate, run before anything else, on MEMENTO single-memory episodes. If patched vectors keep less than roughly the 80–90% of text in-context performance that Geva et al. reported for single-token tasks, then personal memories do not compress into a single hidden state. In that case the project pivots to a diagnostic study of why.
The second check is the main joint-memory comparison against the baselines above. It is run with retrieved keys and with oracle keys, over three seeds, and reports success, the single-to-joint gap, override rate and context tokens. It is decisive only if step-scoped patching beats decomposition with text on the joint gap. The summed-vector ablation must also lose; otherwise the gain has nothing to do with scoping.
The third check is the locality audit. Patching should leave unrelated slots unchanged, in direct contrast to the 52% control retention reported for weight-space edits.
The fourth check is PARSEC transfer, broken down by layout type. We expect the method not to fix Uniform-2D grids. Geometric failures there are outside what a preference vector can carry, and confirming this boundary is part of the result.
References
- Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilization
- TheoremQA: A Theorem-driven Question Answering Dataset
- In-Context Learning Creates Task Vectors
- In-situ Value-aligned Human-Robot Interactions with Physical Constraints
- Personalized Robotic Object Rearrangement from Scene Context
- When Splitting Makes Stronger: A Theoretical and Empirical Analysis of Divide-and-Conquer Prompting in LLMs
- Transformers Are Adaptable Task Planners
- Suppression Sticks, Locality Is Fragile: A Closed-Loop Target-and-Control Audit of Task-Vector Negation in VLA Policies
- Variational Task Vector Composition
- Bi-level Personalization for Federated Foundation Models: A Task-vector Aggregation Approach