Large language models can act as recommenders without retraining. You show them a user's last few interactions as worked examples in the prompt, and they adapt on the fly. This is called in-context learning: the model picks up a task from examples placed in its input, with no weight updates. The approach handles a real problem, which is users drifting in what they want faster than any production model can be retrained. It also has two costs. Every request re-reads a growing block of history text, so latency and memory grow with the history. And the model sometimes names items that are not in the catalog, or recommends real items for no reason it can ground. We propose to replace the history text with a single vector. We run the user's recent interactions through the model once, read off the hidden state the model forms at the end of those examples, and later insert that state into a short query-only prompt. Decoding is restricted to real catalog titles.
The question is whether a "task vector" carries per-user content. Prior work found that such a vector carries a shared rule like "translate to French." A user's taste is closer to data than to a rule, so this is not obvious. If the answer is yes, a recommender can store one small vector per user instead of a long prompt, update it whenever the user acts, and serve recommendations at the cost of a zero-shot call. We will measure throughout whether this compression makes catalog hallucination better or worse.
Core Claim
In-Context Learning Creates Task Vectors showed that a transformer compresses the rule implied by its demonstrations into the hidden state at the last demonstration token, at an intermediate layer (around layer 15 for LLaMA-7B). Patching that state into a forward pass with no demonstrations recovers 80–90% of in-context accuracy, against 10–20% for the bare query. That work studies fixed functions shared across all prompts and single-token outputs. It says nothing about whether the vector carries instance-specific information.
Our claim has three parts. First, for a recommender, the vector extracted from a user's recent interactions encodes that user's current preferences, not just the format "this is a recommendation task." Second, patching it in recovers most of the gain that text demonstrations give over zero-shot prompting. Third, combined with catalog-constrained decoding, it does so without raising the rate of ungrounded recommendations.
The decisive test is a swap. If we patch user A's vector into user B's query, the recommendations should follow A's history rather than B's. A format-only vector, built from shuffled histories, should recover none of the personalization.
Research Program
The work builds on three existing results, each playing a different role. Customizing In-context Learning for Dynamic Interest Adaption in LLM-based Recommendation (RecICL) motivates the setting and constrains the design:
- It adapts a frozen model to drift by inserting the user's most recent interactions as demonstrations.
- It finds that recency-based selection beats semantic similarity.
- It finds that even one recent exemplar gives the largest gain.
- It finds that instruction-tuned models show "inertia" and ignore demonstrations unless first trained on prompts containing them.
RecICL stays at the text level, studies only binary prediction, and leaves item hallucination and prompt length unaddressed. We therefore extract vectors from both a plain instruction-tuned model and a RecICL-style ICL-tuned model. Inertia could make the vector empty in the former.
Avoid Recommending Out-of-Domain Items: Constrained Generative Recommendation with LLMs (RecLM) supplies the grounding method. It uses a prefix tree of valid catalog titles that blocks invalid tokens, and a retrieval variant that matches a hidden state against catalog embeddings. We reuse the prefix-tree constraint as-is, so our contribution is the conditioning signal, not the grounding mechanism.
Tapping the Potential of Large Language Models as Recommender Systems motivates both cost and hallucination concerns. It reports that hallucinated items and position bias are the main failure modes. It also reports that longer contexts do not reliably improve NDCG@10, and that LLM inference exceeds 10 seconds per request against about 1 second for traditional recommenders.
The pipeline works as follows:
- When a user interacts with an item, a background job encodes the user's last $M$ interactions as demonstration text, for example "liked: title". It runs a forward pass through the first $L$ layers only and stores the hidden state $\theta_u$ at the final demonstration token. For a 7B model, this is one 4096-dimensional vector, about 16 KB in half precision. This deferral of heavy work echoes how TiVo moved recommendation computation off the serving path.
- At request time, the model receives a short query prompt. We overwrite the hidden state at the corresponding position and layer with $\theta_u$.
- The model then either scores a candidate (binary prediction, as in RecICL) or generates a next-item title under the prefix-tree constraint.
We sweep $L$ across layers, since the task-vector paper found extraction strongly layer-dependent. We also sweep $M$ from 1 to 20.
Since task vectors are established only for single-token outputs, multi-token title generation is itself a risk. The constrained decoder narrows it: after the first few tokens, the prefix tree leaves few choices. We will report whether the vector's effect is concentrated in the first token of the title.
For hallucination, we measure two quantities:
- Out-of-catalog rate under unconstrained decoding. The constraint makes this zero by construction. Without the constraint, it reveals what the conditioning signal does to the model's own grounding. One plausible view is that text demonstrations help, because the model can copy exact title strings from context. Another is that they hurt, because surface strings prime near-miss variants. Which holds is an open empirical question.
- "Constrained-but-ungrounded" rate under constrained decoding. This is the share of valid-title outputs whose item shares no genre, author, or category with anything in the user's history, broken down by item popularity. Popular items are likely memorized, so we expect failures to concentrate in the long tail.
As an optional analysis, we fit a linear probe distinguishing memorized head items from tail items, following CASAL. CASAL found that "known" and "unknown" facts separate linearly in activations. We ask whether patching shifts tail-item queries toward the "unknown" side.
Baselines
The naive reference is zero-shot prompting: the same frozen model and the same short query, with no history, under the same constrained decoder. It sets the floor that every gain is measured against. The task-vector paper's 10–20% query-only accuracy suggests the floor will be low.
The upper reference is full text in-context learning with the same $M$ recent interactions, selected by recency as RecICL recommends, on both the plain and ICL-tuned models. On Amazon Books, RecICL reports an AUC of 0.8401, against 0.7005 for TALLRec, with a performance gap to a freshly retrained model (its PDM metric) of 0.0031. We treat 0.8401 as the published figure our text-ICL reproduction should approach. We do not expect the vector to beat text ICL in accuracy. Beating it means retaining at least 80% of text ICL's gain over zero-shot, measured as (AUC of patched − AUC of zero-shot) divided by (AUC of text ICL − AUC of zero-shot). We pair that with lower per-request latency and memory, and an unconstrained out-of-catalog rate no higher than text ICL's. The 80% threshold comes from the retention the task-vector paper observed.
The strongest grounded generative competitor is RecLM's retrieval and constrained variants, given the same history text. The file contains no RecLM numbers (its description is truncated), so there is no reported result found on Lacuna for RecLM; we will rerun it on our splits. We also include UNGER from UNGER: Generative Recommendation with A Unified Code via Semantic and Collaborative Integration as the trained generative system. It reports relative gains of 14.68% Recall@20 and 7.85% NDCG@20 over EAGER on Amazon Beauty, but the file gives no absolute values, so we will rerun it rather than quote it. A first-order Markov next-item model, as in Mining Music Playlogs, anchors the cheap end.
All LLM methods share one backbone (LLaMA-family 7B), the same temporal splits, the same candidate sets, and the same decoder constraint. Each gets an equal hyperparameter budget: layer and $M$ for ours, $M$ and exemplar order for text ICL.
Two ablations isolate the key ingredient. The first is a format-only vector, averaged over shuffled histories from other users. If it matches the user's own vector, the method carries no personalization and the claim fails. The second is the cross-user swap.
Why not just…?
Why not just cache the history's key-value states and reuse them? Prefix caching stores the attention keys and values for the history tokens so they are not recomputed. It is the most serious threat on latency, and it is exact rather than approximate. Its weaknesses are memory and update cost. Its memory scales as tokens × layers × 2 × hidden size per user, against one vector. And because the history changes on every interaction, appending to a recency window invalidates positions. We will report memory per user and p50/p95 latency for both. If caching matches our latency at acceptable memory for realistic $M$, the latency argument collapses. The remaining question would then be whether the compressed vector is a better or worse conditioning signal for grounding.
Why not just summarize the history in a sentence of text? A short natural-language interest summary also shortens the prompt. It is a fair competitor at equal token budget. We will generate summaries with the same model and compare them at matched query length. Our expectation is that summaries lose item-level detail that the hidden state retains, but this is exactly what the comparison decides.
Why not just learn a user embedding or soft prompt? Trained per-user embeddings would likely beat a patched vector in accuracy. However, they need gradient updates when tastes shift, which is the retraining cost RecICL was built to avoid. We include a soft-prompt variant updated nightly to show where it wins (stable users) and where the training-free vector wins (users whose last few interactions diverge from their history).
Proposed Experiments
Binary prediction uses RecICL's benchmarks, Amazon Books, Movies, CDs, Sports, and MIND, with AUC and PDM. Generative next-item recommendation uses Amazon Beauty (5-core) and MIND with Recall@20 and NDCG@20, plus the two hallucination rates, each stratified by item popularity.
We use time-based splits throughout. Do LLMs Memorize Recommendation Datasets? raises the concern that models have memorized MovieLens-1M. We therefore run MovieLens-1M only as a contamination check: if the vector "works" there but not on MIND's recent news items, the effect is memorization rather than adaptation.
The decisive checks are four:
- Retention of at least 80% of the text-ICL gain at the best layer.
- A swap test in which patched recommendations correlate more with the donor's history than the recipient's.
- A format-only vector recovering little of the personalization gain.
- An unconstrained out-of-catalog rate, and a tail-item ungrounded rate, no higher than text ICL's.
Failing check 2 or 3 would show that task vectors encode the task and not the user. That would be a clean negative result about what in-context learning compresses.
References
- In-Context Learning Creates Task Vectors
- Customizing In-context Learning for Dynamic Interest Adaption in LLM-based Recommendation
- Avoid Recommending Out-of-Domain Items: Constrained Generative Recommendation with LLMs
- Tapping the Potential of Large Language Models as Recommender Systems: A Comprehensive Framework and Empirical Analysis
- TiVo: making show recommendations using a distributed collaborative filtering architecture
- Hallucination Reduction with CASAL: Contrastive Activation Steering for Amortized Learning
- UNGER: Generative Recommendation with A Unified Code via Semantic and Collaborative Integration
- Mining Music Playlogs for Next Song Recommendations
- Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1M