Recommenders that combine a large language model (LLM) with collaborative filtering, the classic method that predicts what you will like from what similar users liked, now report sizable gains. Those gains are hard to interpret. LLMs have read the public benchmarks they are tested on. GPT-4o can recover the titles of 80.76% of MovieLens-1M movies from their numeric IDs, and it reaches HR@1 of 0.2796 on that dataset against 0.0406 for a standard matrix-factorization model, with the advantage concentrated on popular items (Do LLMs Memorize Recommendation Datasets?). So when a hybrid system beats its collaborative backbone, we cannot tell whether the LLM reasoned about the user's taste or recognized famous titles and popularity patterns it saw during pretraining.
We propose to borrow a tool from privacy research, semantic generalization. Semantic generalization replaces a specific, identifying value with a broader category in a hierarchy until the value no longer singles anyone out, for example turning "16F" into "teenage girl". We apply it to items rather than people. Every item in the user's history and in the candidate set is rewritten as the most specific description that still fits at least $k$ items in the catalog. We call these $k$-ambiguous descriptions. The LLM can still reason over genre, era, tone, and themes, but it cannot look up the title it would otherwise recognize. By sweeping $k$ we get a dial that removes the memorization channel and leaves the content channel, so we can measure how much of a hybrid's gain survives once the model can no longer name what it is looking at.
Core Claim
The claim is that $k$-ambiguous item abstraction separates the LLM's contribution to a hybrid recommender into a recognition part and a reasoning part. We make one testable prediction: the LLM's gain over a collaborative filtering baseline shrinks with $k$ much faster on popular items than on long-tail items. Write $\Delta_g(k)$ for the gain in NDCG@10 of the hybrid over its collaborative backbone on item-popularity group $g$ when items are shown at ambiguity level $k$. The prediction is $\Delta_{\text{head}}(k) \to 0$ at moderate $k$, while $\Delta_{\text{tail}}(k)$ stays clearly positive. The prediction follows from the memorization paper's finding that recall is much stronger for the top 20% of items (89%) than the bottom 20% (64%).
If the prediction holds, there is a practical consequence. A hybrid that feeds the LLM abstracted descriptions and routes weight to the LLM only where abstracted reasoning helps should match raw-title hybrids on fair, contamination-controlled evaluation. If $\Delta$ collapses everywhere, the published gains of LLM hybrids on public benchmarks are largely leakage, which is an equally important result. If $\Delta$ survives everywhere, memorization is not the driver, which is reassuring evidence for the field.
What is new is the instrument. We control how identifiable an item is with a catalog-level ambiguity guarantee, and we use that control both to audit hybrids and as the input representation of a hybrid.
Relation to Prior Work
The gap is motivated by two audits. Do LLMs Memorize Recommendation Datasets? shows title and popularity memorization on MovieLens-1M but offers no way to evaluate around it. The seed audit More Context Is Not Validation shows that richer textual context can fail to improve, or can worsen, LLM predictions of user behavior: historical content digests worsened Brier error by +0.0129 relative to a raw-candidate arm, and a calibrated prior (Brier 0.216) beat the grounded LLM hybrid (0.235). That paper's baseline-first stance is the standard we adopt. We do not have its listener data, so we work on public benchmarks where contamination is the documented risk.
The hybrid methods we audit share one feature: they hand the LLM literal titles. Llama4Rec fuses a conventional recommender with an instruction-tuned LLM, shifts weight toward the LLM for sparse users using a long-tail coefficient, and reports a 14.65% relative gain in direct recommendation on MovieLens and BookCrossing. DELRec distills a sequential recommender's patterns into soft prompts for a fine-tuned LLM. BridgeCRS re-ranks GPT-3.5 suggestions with collaborative embeddings learned by SASRec, a standard transformer-based sequential recommender. We keep their fusion designs and change only what the LLM sees. CESRec cleans histories by masking outlier items entirely, and performance peaks at two masked items. Our method generalizes items instead of deleting them, and its purpose is controlling leakage, not removing outliers.
The method comes from privacy work. NAP2 shows that obscuring a detail by rewriting it more abstractly keeps text usable where deletion and differential-privacy noise do not: a 93.81% privacy-entailment score for the fine-tuned rewriter, while differential-privacy methods score below 63%. Heatherly et al. generalize rare clinical codes up a hierarchy so that statistical associations survive while patients cannot be re-identified. The social-media publishing middleware enforces $k$-anonymity, the guarantee that every released record matches at least $k$ people, on free text. Our $k$-ambiguity is the item-side analogue of that guarantee. MaskSQL abstracts database values before querying a cloud LLM, which is the closest idea in spirit, but it targets privacy rather than measurement. Synonymix merges individual personas into group-level abstractions that keep behavioral signal (r = 0.59 on General Social Survey items) while capping any one source's contribution. It shows that abstraction can preserve signal, but nobody has tested it inside a recommender, and we do not depend on it.
Research Program
We first build an item abstraction hierarchy for each dataset. Level 0 is the title. Level 1 is a title-free description written by an LLM from structured metadata: year, genres, creators, and a plot or product description paraphrased so that it contains no names. Level 2 is a bundle of attributes: decade, genre set, tone, and theme tags. Level 3 is the coarse category alone. For each item we also compute an ambiguity count, the number of catalog items consistent with its description. For structured levels this is an exact match count. For free text we count catalog items that a retrieval model plus an LLM judge accept as consistent. The abstraction operator takes a target $k$ and returns, for each item, the most specific level whose ambiguity count is at least $k$, using per-field coarsening where needed, in the same way hierarchical $k$-anonymization works.
We do not trust the guarantee on paper; we test it empirically. For every item and level we ask each evaluated LLM to name the item from its description and record the identification rate. This rate is our direct measure of the leakage channel, and we report it next to every accuracy number.
We then rerun representative hybrids with items shown at $k \in \{1, 5, 20, 100\}$ plus the raw title, keeping everything else fixed. The hybrids are a Llama4Rec-style adaptive fusion, a BridgeCRS-style re-ranker, and a simple late fusion of LLM scores with SASRec scores. Finally we build the deployable variant. It is a late-fusion hybrid whose LLM input is abstracted at a $k$ chosen on validation data, and whose fusion weight depends on item popularity and user history length. It is meant to be a modest, honest system, not a new architecture.
Baselines
The naive reference point is a calibrated popularity prior: item popularity in the training split, calibrated with grouped cross-validation over users. This mirrors the listener-grouped prior that beat the LLM hybrid in More Context Is Not Validation (Brier 0.216 versus 0.235 on that paper's cohort, which is not comparable to ours). Any hybrid that fails to beat it on tail items has no claim.
The collaborative backbones are BPRMF, a standard matrix-factorization model trained to rank items a user interacted with above ones they did not, and SASRec. On MovieLens-1M, Do LLMs Memorize Recommendation Datasets? reports HR@1 of 0.0406 for BPRMF against 0.2796 for GPT-4o. We found no reported absolute SASRec result on Lacuna for MovieLens-1M under our protocol, so we will measure it.
The strongest existing competitor is the raw-title Llama4Rec-style hybrid (Llama4Rec). The file reports only relative gains for it (14.65% direct, 14.21% sequential), not absolute numbers, so we reimplement it. All systems share the same splits, which are leave-last-out and additionally time-based. They share the same sampled candidate sets, the same LLMs (GPT-4o, plus open-weight 8B and 70B models whose pretraining cutoffs we record), the same LoRA fine-tuning budget where fine-tuning applies, and equal hyperparameter-search trials per method. Beating Llama4Rec means one of two things on the abstracted evaluation: matching its tail-item NDCG@10 within the bootstrap interval while keeping title identification below 5%, or exceeding it on tail items outright.
Two ablations isolate the key ingredient. The first is a title-stripped control: rich descriptions with the name removed but no $k$ constraint, which tests whether removing the name is enough or whether ambiguity is what matters. The second is CESRec-style masking of the identifying fields, which tests whether generalizing beats deleting.
Why not just…?
Why not just evaluate on items released after the model's training cutoff? This is the most natural contamination control, and we include it as a second, independent check on recent Amazon data. The weakness is that post-cutoff items are newer, sparser, and less popular, so this check confounds "the model has not memorized it" with "the item is in the tail." That is exactly the comparison we need to keep separate. Abstraction holds users, items, and popularity fixed and changes only identifiability. If both controls give the same picture of $\Delta$, the result is robust. If they disagree, the difference tells us how much of the post-cutoff effect is a popularity shift.
Why not just add the category features directly to a collaborative model and skip the LLM? This is a real threat. If abstracted descriptions help mainly because they carry genre and era, then a factorization model with the same attribute features, or a gradient-boosted model fusing CF scores with those attributes, may capture all of it more cheaply. We run that model with exactly the attributes the abstraction exposes at each level. If it matches the abstracted LLM hybrid, we will report that the LLM adds nothing beyond its features once leakage is removed, and the deployable variant becomes unnecessary.
Why not just use a well-tuned collaborative model or calibrated prior and drop LLMs? On head items this may well be the right answer, and our prediction says so. The open question is the tail, where LLM-side reasoning is supposed to pay off. The head-versus-tail split of $\Delta(k)$ answers it directly instead of averaging the two groups together.
Proposed Experiments
The primary benchmark is MovieLens-1M, the documented case of contamination. Because the dataset itself has only title, year, and genres, we attach external metadata through title–year matching and report the match rate. We add Amazon-Book and Amazon-Music from the Think before Recommendation evaluation, which have native descriptions and categories, and Reddit-ML32M from BridgeCRS for the conversational re-ranking setting. Ranking uses one held-out positive among 99 sampled negatives, plus a full-catalog evaluation for the collaborative backbones. Metrics are HR@10, NDCG@10, and Brier score on calibrated click probabilities, each reported separately for the head (top 20% by popularity), middle, and tail (bottom 20%), with user-level bootstrap intervals.
The first decisive check is the identification curve. For $k \geq 20$, title identification should fall below 5% for all models. If it does not, the abstraction is leaking and the remaining results are uninterpretable until we fix it.
The second is the shape of $\Delta_g(k)$. A head-item gain that collapses while the tail-item gain persists would support the central claim. A uniform collapse would mean the reported hybrid gains on these benchmarks are mostly recognition. A uniform persistence would mean they are mostly reasoning.
The third is the comparison with the feature-matched non-LLM model at each $k$, which settles whether the LLM adds anything beyond the attributes it is shown.
The fourth repeats the head/tail analysis on post-cutoff Amazon items, which tests whether the abstraction result and the temporal result agree.
Together these produce either a contamination-controlled evaluation protocol that hybrid recommender papers can adopt, or a clear negative finding about what current LLM hybrids actually contribute.
Workshop authors to connect with
- More Context Is Not Validation: A Baseline-First Audit of Action-Grounded LLM Listener Agents: Chinmay Rawat, Taaha Kazi, Vasu Sharma
- Synonymix: Unified Group Personas for Generative Simulations: Huanxing Chen, Aditesh Kumar
References
- Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1M
- More Context Is Not Validation: A Baseline-First Audit of Action-Grounded LLM Listener Agents
- Integrating Large Language Models into Recommendation via Mutual Augmentation and Adaptive Aggregation
- DELRec: Distilling Sequential Pattern to Enhance LLMs-Based Sequential Recommendation
- Bridging Conversational and Collaborative Signals for Conversational Recommendation
- CESRec: Constructing Pseudo Interactions for Sequential Recommendation via Conversational Feedback
- NAP2: A Benchmark for Naturalness and Privacy-Preserving Text Rewriting by Learning from Human
- Enabling Genomic-Phenomic Association Discovery without Sacrificing Anonymity
- A Privacy Preserving Data Publishing Middleware for Unstructured, Textual Social Media Data
- MaskSQL: Safeguarding Privacy for LLM-Based Text-to-SQL via Abstraction
- Synonymix: Unified Group Personas for Generative Simulations
- Think before Recommendation: Autonomous Reasoning-enhanced Recommender