Is It Recommending or Remembering? Counterfactual Catalogs for Auditing LLM Recommenders

Recommenders built on large language models, also called generative recommenders, now outperform classic ID-based sequence models on the standard Amazon and MovieLens benchmarks. A generative recommender treats "what will this user buy next" as text generation over item titles or item tokens. These benchmarks are old, public, and heavily crawled. Product titles, co-purchase lists ("customers who bought X also bought Y"), and movie pairings sit throughout the web text these models were pretrained on. Some of the reported gains may therefore come from the backbone recalling item associations it read during pretraining rather than learning user behavior from the training split. Nobody in the generative recommendation literature has measured how much. We propose to measure it directly. We build counterfactual catalogs: the same users and the same interaction sequences, but every item's text is rewritten from its attributes so that it keeps its meaning and loses the surface strings a model could have memorized. Recommenders are then retrained and re-scored on these catalogs.

The idea in two sentences is this. If a text-based recommender's advantage over an ID-only model survives when item text is rewritten into unmemorizable but meaning-preserving form, the advantage is real semantic generalization. If it shrinks, part of it was recall of pretraining data. A small team of role-separated language-model agents produces the rewrites. The rewriting agent never sees the original text, and an adversary agent tries to re-identify each item. That separation is what makes the rewrites trustworthy, and the proposal tests it explicitly rather than assuming it.

Core Claim

We claim that the text-derived portion of generative recommendation accuracy can be split into a part that transfers to meaning-preserving counterfactual text and a part that does not. We expect the non-transferring part to be largest where web memorization is most plausible: famous movies in MovieLens-1M, older Amazon categories, and compact backbones specialized on item titles. Because the interaction data are held fixed, an ID-only sequence model scores identically on the original and counterfactual catalogs. This gives a clean anchor. We summarize the result with one number, the memorization share

$$\rho = \frac{M_{\text{orig}} - M_{\text{cf}}}{M_{\text{orig}} - M_{\text{ID}}},$$

where $M_{\text{orig}}$ is a text-based recommender's Recall@10 on the original catalog, $M_{\text{cf}}$ is its Recall@10 after retraining on the counterfactual catalog, and $M_{\text{ID}}$ is the Recall@10 of an ID-only model such as SASRec on the same split. A value of $\rho$ near 0 means the text advantage is genuine. A value near 1 means the text advantage over pure collaborative filtering is almost entirely tied to memorizable strings.

The motivation comes from contamination research outside recommendation. Snoopy showed that few-shot accuracy tracks how often evaluation terms appear in The Pile. Studying and improving reasoning in humans and machines found near-zero surprise on classic puzzles, which suggests models recall them rather than solve them. DyVal reported that models strong on static benchmarks "scored nearly zero" on freshly generated instances. DyVal also concedes that such dynamic generation has been confined to formal domains. We carry the dynamic-evaluation idea into recommendation. We do not generate new logic problems. Instead we regenerate the surface of a real behavioral dataset while leaving its behavioral structure untouched. This keeps real user behavior, which synthetic preference worlds would lose, and it removes the memorization channel.

What Is Being Audited

The subjects are the three dominant ways of putting items into a language model. LLM2Rec fine-tunes a small backbone (down to Qwen2-0.5B) to predict next-item titles, then converts it into a bidirectional embedding model. It reports an average 15% relative gain over generalist embedders such as BGE and GTE. The authors attribute this gain to the model learning collaborative signal. A contamination reading attributes it to fine-tuning reactivating title co-occurrences the model already saw on the web. Our experiment separates these two readings. CoVE adds one token per item to the vocabulary, initialized and trained with the item's text in context. CoderRec fuses semantic IDs from an RQ-VAE, a learned quantizer that maps item text embeddings to short discrete codes, with native collaborative IDs. Its ablations already show that native IDs cannot be dropped, which hints that semantic text alone is weaker than headline numbers suggest. Semantic-ID generation in the style of the GRID handbook completes the set. We expect these representations to differ in exposure. Title generation should be most exposed to memorization. Semantic IDs should be least exposed, because quantization discards surface strings. That ordering is itself a testable prediction.

The specialization direction enters here. Specializing Smaller Language Models towards Multi-Step Reasoning shows that compact models concentrate capacity on the target domain and lose ability elsewhere. We ask the recommendation analogue: when a 0.5B–7B model is specialized on item titles, how much of the gain is portable domain knowledge, and how much is the pretraining index of those particular titles? We run backbones of several sizes. This tests whether $\rho$ grows with model size, which it should if larger models memorize more of the catalog text.

Research Program

The first component is a zero-training contamination probe. Before any fine-tuning, we give each base backbone a user's history as a list of original titles and ask it to generate the next title. We score exact and fuzzy matches against the held-out item. We repeat this with the counterfactual text. A base model that already predicts held-out Amazon co-purchases or MovieLens sequences well above popularity has memorized association structure. The drop from original to counterfactual text in this setting gives a lower bound on exposure that does not depend on any training recipe.

The second component is the counterfactual catalog generator, which is where multi-agent decomposition enters as an instrument. Four roles run as separate model instances with deliberately limited information. An extractor reads the original item text and outputs a structured attribute record: category, function, materials, genre, era, price band, audience. It removes proper nouns, brands, and titles. A composer sees only this record and writes a new title and description. Because it never sees the original, it cannot copy memorizable phrasing. An adversary sees the rewritten text plus a pool of candidate originals and tries to re-identify the item. We use a strong generalist model and web-scale search-style matching over the full catalog. A judge checks whether the rewrite is faithful by re-extracting attributes from it and comparing them with the record. Items that the adversary re-identifies or the judge rejects go back to the composer, with a bounded number of retries. Remaining failures are dropped and reported.

This design borrows the information-isolation logic of the tool-separated agents in LLM-RDF and the critique loops of MACRec and ExeGen. It uses them for a different job. Those systems decompose the recommendation itself. We decompose the construction of an evaluation, following Benchmark Self-Evolving, which used multiple agents to evolve reasoning benchmarks, but here the target is recommendation catalogs. Note that the generator is not the recommender. Its latency does not matter, which sidesteps the per-request cost that MACRec itself acknowledges.

We also build two controls from the same data. A shuffled catalog permutes the rewritten texts across items. This destroys any link between text and behavior and gives the floor for "text carries no signal." An attribute-only catalog renders the extractor's record as a templated string with no prose, which isolates how much of the text signal lives in structured attributes.

The third component is retraining and scoring. Every recommender is trained from its public recipe on the original, counterfactual, shuffled, and attribute-only catalogs. Interactions, splits, compute, and hyperparameter budgets are identical across catalogs. We report Recall@10, NDCG@10, and HR@5, plus $\rho$ per method, dataset, and backbone size. We also stratify results by item popularity and by the adversary's pre-rewrite confidence that the item is famous. This checks whether any gap concentrates on web-visible items.

Baselines

The naive reference point is a popularity ranker. KuaiRec shows that popularity heuristics look artificially strong under sparse, biased evaluation. We report popularity so that no text-based method's counterfactual score is mistaken for progress when it only matches popularity.

The anchor baseline is SASRec trained on item IDs alone. It is the strongest competitor on the specific question "is the text helping at all," and it is invariant to the catalog rewrite by construction. We run it with the GENPAS augmentation from Sequential Data Augmentation for Generative Recommendation, using the snap-research/GenPAS code, because that paper reports gains "up to 783.7% in NDCG@10" from data construction alone. A weakly tuned ID baseline would inflate the apparent text advantage and make $\rho$ uninterpretable. We found no reported absolute SASRec number on these splits in the file, so we reproduce it ourselves with the same tuning budget as every text model.

The strongest text-based methods are CoderRec, CoVE, and LLM2Rec. CoderRec reports R@10 of 0.0830 on Amazon Beauty, the highest among the methods it tested (Cross-Scale Collaboration). CoVE reports up to a 62% boost in NG@5/HR@5 on Sports. LLM2Rec reports a 15% average relative gain over generalist embedders on Games and Arts. We first reproduce each result on the original catalog to within a stated tolerance, and only then retrain on counterfactual catalogs. A method "beats" contamination in our sense if its counterfactual score keeps most of its margin over tuned SASRec, operationally $\rho < 0.25$ with a bootstrap interval that excludes 0.5. A method with $\rho$ above 0.5 on a dataset should not have that dataset cited as evidence that its item representation learns behavior.

The ablation that isolates our key ingredient is the generator itself. We compare the full isolated pipeline against a single model that paraphrases the original text directly under the same instruction to remove names and brands. Both are measured on adversary re-identification rate and judge fidelity.

Why not just…?

Why not just use a temporal split with items released after the backbone's pretraining cutoff? This is the most serious alternative, and we run it on Amazon 2023 categories as a companion experiment. It has a confound. Post-cutoff items are also newer, colder, and less popular, so a drop in accuracy conflates memorization with cold-start difficulty. To the Cutoff... and Beyond? documents post-cutoff degradation but cannot say why it happens. Counterfactual catalogs keep each item's popularity and history fixed and change only its memorizability. If the temporal and counterfactual estimates of $\rho$ agree, the cheaper temporal test is validated for future use. If they disagree, the gap measures the cold-start confound.

Why not just replace item text with random strings or hashes? Doing so removes memorization and meaning together, so any drop mixes "lost memorized associations" with "lost semantics." That comparison is essentially the ID-only baseline, which we already run. The shuffled and attribute-only controls, together with the meaning-preserving rewrite, are what separate the two causes.

Why not just run membership-inference or perplexity tests on item titles? Low perplexity on titles shows that the model saw them. It does not show that seeing them drives recommendation accuracy, because a model can memorize a title without memorizing which items co-occur with it. We do compute title perplexity. We use it as a covariate, predicting per-item accuracy drops from it, not as the verdict.

Why not just have one LLM paraphrase the descriptions? This is plausibly sufficient, and the ablation above decides it. Our expectation is that a paraphraser that sees the original will leak distinctive phrases and named entities at a measurably higher re-identification rate. The composer, which only sees attributes, cannot leak what it never saw. If the single paraphraser matches the pipeline on both leakage and fidelity, we will report that and use it, because the multi-agent structure is justified only by that measured difference.

Proposed Experiments

We cover five regimes chosen to span contamination risk. MovieLens-1M has iconic titles and should show the highest risk. Amazon Beauty, Sports, and Toys are old, heavily crawled categories. Amazon 2023 Games and Arts are newer. Goodreads serves as LLM2Rec's out-of-domain transfer test. KuaiRec has short-video captions that are unlikely to appear in English web corpora, and its fully observed small matrix removes the exposure-bias confound that KuaiRec itself documents. It acts as a low-contamination control where $\rho$ should be near zero. Backbones range from 0.5B to 7B within one model family, so that size effects are not confounded by family.

Three checks decide the outcome. First, the generator must achieve adversary re-identification near chance and attribute fidelity high enough that the attribute-only control on the counterfactual catalog matches the attribute-only control built from original records. That condition confirms the rewrite preserved the information the rewrite was allowed to keep. Second, $\rho$ should be ordered by expected exposure: MovieLens above old Amazon above Amazon 2023 above KuaiRec. It should also be ordered by representation, with title generation above vocabulary expansion above semantic IDs. Third, $\rho$ should rise with backbone size, as it should if memorized catalog knowledge scales with capacity.

If all three hold, the field gains a routine audit and a corrected picture of which generative recommenders learn behavior. If $\rho$ is near zero everywhere, that is also a useful result: it would be the first direct evidence that text-based recommenders' gains are not a contamination artifact, and it would answer the tension raised by LLM2Rec's claim.