Recommenders that generate their answer are increasingly used to predict what a user will want next. These are language models that read a user's history and write out the next item, spelled either as its title or as a short sequence of learned "semantic ID" codes. Such models have no way to say "that one" about an item already sitting in their input. The input may be a past purchase the user is likely to repeat, or a new item placed in the prompt by a retriever. In either case the model must regenerate the item's code sequence one token at a time from its own parameters. This fails most often for the items where pointing would help most: brand-new items whose codes the model never learned to produce, and items buried in histories full of accidental clicks.
We propose a pointer-generator recommender whose unit of pointing is the whole item rather than the individual token. A pointer-generator is a decoder that mixes writing from its vocabulary with copying spans from its input. At each step, our model decides how much probability to put on generating a new item and how much to put on selecting one of the items in its context. The selection scores come from a head trained to ignore distractor items that were injected on purpose, not from raw attention. The core claim is testable. Token-level copying over semantic codes produces "chimera" items, and attention-based copying is poisoned by noisy histories. Copying whole items with an explicitly trained distractor signal should recover cold-start and repeat items without losing accuracy on novel ones.
Core Claim and Why the Obvious Version Fails
The obvious combination would bolt the classic copy gate of Get To The Point onto a semantic-ID recommender such as LETTER. LETTER tokenizes items into hierarchical code tuples with a residual-quantized autoencoder (RQ-VAE, a model that compresses each item's features into a short sequence of discrete codes). The classic gate mixes vocabulary probability with attention weights over individual input tokens. Applied to code tuples, it can copy the first code from one history item and the second code from another. The result is a valid-looking but wrong or nonexistent item.
This is the recommendation analogue of a problem in domain-specific tokenization. Work on vocabulary adaptation such as AVocaDo and MedTPE shows that splitting a domain concept into generic pieces dilutes its meaning and forces the model to reassemble it. We take the lesson from tokenization, that the domain concept should be the atomic unit, and apply it to the copy mechanism rather than the vocabulary.
The second failure is noise. Both LargePiG and the classic pointer-generator derive the copy distribution from soft attention. LargePiG builds a training-free pointer-generator from an LLM's own attention maps and a layer-divergence copy gate. Research on contextual distraction shows that soft attention spreads over irrelevant context. DAS documents reader collapse as distractor documents grow to 100. Ext2Gen shows that extraction learned jointly with generation beats separate compression under up to 25 distractor chunks. These results suggest that attention-derived pointers will copy accidental clicks. Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That! makes the same point from translation: an uncontrolled copy mechanism copies what it should not. These papers motivate the design and also constrain it: the pointer cannot simply read attention.
Research Program
The model is a semantic-ID generative recommender in the style of LETTER or Purely Semantic Indexing. The latter resolves collisions where two items receive the same codes by searching for the next-best semantic centroid instead of appending arbitrary counters. We use that indexing so every item has a unique, meaningful code tuple.
The context contains the user's history items and, optionally, a candidate list from a text-based retriever in the spirit of UniSRec. UniSRec encodes item text with a frozen BERT, whitens the embeddings, and matches sequences to items without shared IDs. The candidate list lets cold-start items appear in the context even though the generator has never been trained to produce their codes.
We pool the hidden states over each context item's code span into one vector per item. The next-item distribution over the catalog $\mathcal{I}$ is then
$$P(v) = g\, P_{\text{gen}}(v) + (1-g)\, \frac{\exp(s_v)}{\sum_{u \in C} \exp(s_u)}\,\mathbb{1}[v \in C].$$
Each symbol is defined as follows. $P_{\text{gen}}(v)$ is the product of code-token probabilities under prefix-trie constrained decoding, which restricts decoding to valid item codes. $C$ is the set of items in context. $s_v$ is a score from a small head over the pooled item vector and the decoder state. $g \in [0,1]$ is a learned gate computed from the decoder state.
The key training ingredient is distractor injection. With some probability during training, we insert random items into the history, either same-category or cross-category, and hard-negative items into the candidate list. We add a loss that pushes the scores $s_v$ of injected items toward zero pointer mass, because the labels for which items are distractors are known by construction. This follows Ext2Gen's idea of learning extraction jointly with generation. It operates on atomic items instead of sentences and needs no preference data.
The analysis half of the program measures three things. First, pointer mass on injected distractors as a function of noise level. Second, the rate of chimera and invalid items under token-level copying. Third, how the gate $g$ behaves on repeat, novel, and cold-start targets.
Baselines
The naive reference points are recommending the user's most recent item and recommending the globally most popular items. On repeat-heavy data such as Amazon Beauty, the last item is a nontrivial baseline for a copy mechanism.
SASRec, a standard ID-based sequential recommender, is the discriminative reference. No reported result was found on Lacuna for SASRec on these splits, so we will run it ourselves.
The strongest existing generative method is Purely Semantic Indexing. It reports NDCG@5 of 0.2702 on Amazon-Beauty, against 0.2611 for its vanilla semantic-ID baseline (Purely Semantic Indexing). It is the strongest competitor because it already fixes collisions and improves cold-start, which is exactly where we claim gains. We reuse its indexing, backbone, split, and tuning budget. The only difference is the item-level copy path, so any gap is attributable to copying. Beating it means exceeding its NDCG@5 on clean Beauty histories, and showing a flatter degradation curve as injected noise grows. LETTER is run the same way; no reported number was found on Lacuna for its Recall@10 or NDCG@10.
For title-generating models we include GenRec. GenRec fine-tunes LLaMA to write the next item's title. It reports HR@5 of 0.1034 on MovieLens 25M and 0.0190 on Amazon Toys, below P5's 0.0239 on Toys. Toys is the regime where item names are fragmented and semantically sparse, and it is where item-level copying should matter most.
Three ablations isolate the proposal's key ingredients. Token-level copy applies the classic pointer-generator over code tokens and tests the atomic-unit claim. Attention-derived item pointing aggregates LargePiG-style attention over each item's span instead of using a trained head, and tests whether distractor supervision is needed. The model without distractor injection tests the noise objective itself. All variants share data, compute, and hyperparameter search budget.
Why not just…?
Why not just use trie-constrained decoding, or rerank the candidate list by generative likelihood? Constrained decoding already guarantees valid items, and reranking candidates by $P_{\text{gen}}$ also "selects from context." This is the real threat. Reranking cannot recommend outside the candidate list. More importantly, for cold-start candidates $P_{\text{gen}}$ is low for exactly the reason we care about: their codes were never trained as targets. We will settle this by sweeping retriever recall by varying candidate list size and retriever quality. We report reranking, pure generation, and our mixture on cold-start and warm targets separately. If reranking matches the mixture at every recall level, the copy path adds nothing, and we would report that.
Why not just filter the history before the recommender sees it? One could score history items for relevance with a DAS-style query-likelihood score or a separate compressor and drop low scorers. Ext2Gen found that independent compressors such as Recomp, CompAct, and EXIT underperform joint extraction and generation. It reports an average QA score of 0.557 against 0.473 for the default Llama3.1-8b model. That result is from question answering, not recommendation, so we run a pre-filter baseline with the same scoring head trained separately. A hard filter also discards weak but real signal, which the soft mixture keeps.
Why not just give every item its own vocabulary token? Atomic item tokens, initialized by averaging the subword embeddings as IndoBERTweet does for new words, make copying a single-token operation. However, this discards semantic sharing across items and gives cold-start items untrained embeddings. That is the same weakness Purely Semantic Indexing found with non-semantic suffixes. We include this variant on Instruments, which has the smallest catalog, as a cold-start comparison.
Proposed Experiments
We use Amazon Beauty, Toys, and Instruments, Yelp, and MovieLens 25M with leave-one-out splits. We report Recall@10, NDCG@10, HR@5, and NDCG@5, stratified by target type: repeat, novel-warm, and cold-start.
Noise is evaluated in two forms. Synthetic noise injects $k \in \{0, 2, 5, 10, 20\}$ random items. Naturalistic noise merges a user's histories across Amazon categories so that off-category purchases serve as distractors.
Three checks are decisive. First, token-level copy should show a measurable chimera rate that item-level copy eliminates. Second, pointer mass on injected distractors should stay near zero with distractor supervision and rise with $k$ without it. Third, on cold-start targets under realistic retriever recall, the mixture should beat both pure generation and candidate reranking. If the third check fails, the claim reduces to a noise-robust repeat-purchase model. That is still useful, but it is narrower than the proposal, and we will state it that way.
References
- Get To The Point: Summarization with Pointer-Generator Networks
- Learnable Item Tokenization for Generative Recommendation
- AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain
- From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Prediction
- LargePiG: Your Large Language Model is Secretly a Pointer Generator
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Pointer-Generator Networks for Low-Resource Machine Translation: Don’t Copy That!
- Purely Semantic Indexing for LLM-based Generative Recommendation and Retrieval
- Towards Universal Sequence Representation Learning for Recommender Systems
- GenRec: Large Language Model for Generative Recommendation
- IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization