Recent LLM recommenders split the job across several cooperating language-model "agents": one summarizes the user, one judges candidate items, one ranks them. They also feed these agents retrieved records, such as the user's interaction history and item metadata from the catalog. They report large gains over classical recommenders. These systems rest on an untested assumption: that the agents reason over the records they are handed. An LLM has also memorized a great deal about popular movies and products, and on public benchmarks it can often skip the records and rely on that memory. We call this failure parametric substitution. An agent replaces what the retrieved record says with what the model remembers. Once that claim is written into the shared workspace the agents read from, every later agent treats it as fact.
This proposal does two things. First, it builds a counterfactual-catalog test in which item records deliberately contradict what an LLM is likely to remember, so the test can tell whether a recommender follows its evidence. Second, it builds and evaluates claim-checked handoffs. Every message one agent passes to another must state its factual claims as (item ID, attribute, value) triples, and a plain database lookup, not another LLM, verifies each triple before the message is admitted. The central question is whether checking at every handoff, rather than only at the final output, is what stops memorized errors from cascading. That question also decides whether multi-agent decomposition earns its cost in this setting.
Core Claim
Hallucination in LLM recommenders is usually framed as inventing items that do not exist. Restricting outputs to valid catalog IDs already largely handles this, as ToolRec's memory strategy shows by matching candidates against the item directory (Let Me Do It For You). We argue the harder and less-measured failure is subtler: the system recommends real items for remembered rather than retrieved reasons, and fabricates or overrides attributes along the way. Two predictions follow. Part of the reported gain of agentic recommenders over sequential models comes from memorization and will shrink when titles and attributes stop matching the model's memory. A deterministic check at each inter-agent handoff will recover faithfulness to the records at little cost in ranking quality, and will do better than a single check on the final answer.
Grounding in Prior Work
The motivation comes from three findings in the file. Do LLMs Memorize Recommendation Datasets? shows GPT-4o reproduces 80.76% of MovieLens-1M titles. It reaches HR@1 of 0.2796 against 0.0406 for BPRMF, and it recalls popular items far better than tail items (89% vs 64%). This establishes that memory alone can produce benchmark wins. Holt et al. (Samuel Holt et al.) describe how a sampled error conditions all later generation. Adaptive Memory Admission Control For LLM Agents shows that storing unverified outputs corrupts later agent reasoning. We borrow its idea of gating what enters memory and apply it to inter-agent messages rather than long-term storage.
The systems we test against are the closest existing work. MACRec splits recommendation into a Manager, Analysts, a Searcher and a Reflector. The Reflector is an LLM that critiques the final output. ARAG runs a user-understanding agent, a natural-language-inference agent, a summarizer and a ranker over a shared workspace. It reports up to 42.12% NDCG@5 improvement on Amazon Clothing, but it does not measure attribute hallucination. Both verify with LLM judgment, which inherits the overconfidence that cross-examination studies expose (Ayush Agrawal et al.). Neither controls for memorization. Our proposal differs in three ways. Verification is a database lookup rather than a model opinion. It happens at every handoff rather than at the end. Evaluation uses records designed to conflict with memory.
Two methods constrain the design. MAIN-RAG filters retrieved passages using a judge agent's yes/no log-probabilities with an adaptive threshold. It is a strong LLM-based filter, and we run it as the learned-judge alternative to our lookup. LCFT makes a fine-tuned recommender give consistent answers to opposite prompts, but logical consistency does not ensure agreement with the catalog. That gap is exactly what a lookup closes. RETURN purifies noisy user histories with a co-occurrence graph and shows LLM recommenders are fragile to small profile changes. Our counterfactual perturbations target the item side rather than the history side, so the two are complementary.
Research Program
The first component is the counterfactual catalog. We start from MovieLens-1M and from the Amazon Review 2016 Clothing, Electronics and Home & Kitchen subsets that ARAG uses. From these we build three conditions. The original condition is unchanged. The masked condition replaces titles and brand names with opaque identifiers but keeps descriptive metadata and the interaction structure, so collaborative signal survives and title memory does not help. The conflict condition swaps a controlled fraction of salient attributes (genre, brand, material, release period) between items. Retrieved records then contradict plausible memory. For conflict items we know which answer is record-faithful and which is memory-driven.
The conflict condition also gets a conversational variant built from the same catalog. A user states a hard constraint, such as "no items from brand X". The record says a candidate violates the constraint and memory says it does not, or the reverse. This gives a direct measure of whether the system obeys its evidence.
The second component is the claim-checked handoff protocol. Each agent's output has two parts: free-text reasoning, and a structured list of claims as (item ID, attribute, value) or (user ID, interacted item ID) triples. Only the structured claims, plus reasoning text that refers to verified claims, are passed downstream. A deterministic checker queries the catalog and the interaction log. Claims that contradict the records are rewritten to the recorded value and flagged. Claims that cannot be verified are dropped. The rate of each is logged per agent, which makes cascades visible: we can see which upstream agent introduced an error and whether downstream agents repeated it.
We apply the protocol to a re-implementation of the ARAG pipeline so the agent roles are held fixed. The only change is the handoff discipline. Final explanations go through the same checker.
Baselines
The naive reference points are a popularity ranker and SASRec trained on each split. These require no LLM, so they cannot be affected by memorization. The file contains no reported SASRec numbers on these exact splits. The closest classical-plus-LLM figure is CoderRec's R@10 of 0.0830 on Amazon Beauty (Cross-Scale Collaboration), which we cite only as context because Beauty is not among our splits. A single-LLM vanilla retrieval baseline, with the same candidate pool and records in one prompt, represents the monolithic alternative.
The strongest competitor is ARAG. It is the published agentic retrieval recommender with the largest reported gains, and it operates on the same Amazon subsets. Its reported result is relative: 42.12% NDCG@5 and 35.54% Hit@5 improvement on Clothing, 37.94% NDCG@5 on Electronics and 25.60% on Home & Kitchen, each over its retrieval baselines (ARAG). The file reports no absolute NDCG values for ARAG, so we reproduce it and report our own absolute numbers. We also run MACRec with its Reflector; no quantitative benchmark result for MACRec is reported on Lacuna. We run ToolRec-style catalog-ID filtering as the "just block nonexistent items" defense.
For fairness, every LLM-based system uses the same open-weight base model and the same initial vector-retrieval candidate pool of fixed size. Each gets the same per-query token cap and the same prompt-tuning budget, measured as a fixed number of development-set iterations. Seeds and calibration splits are shared.
Beating ARAG means the following. On the conflict condition, we need higher record-faithful ranking agreement and higher claim precision with non-overlapping confidence intervals. On the original condition, NDCG@5 must be within one standard error of ARAG.
Three ablations isolate the key ingredient. The first applies the checker only to the final output. The second replaces the lookup with a MAIN-RAG-style log-probability judge at each handoff. The third collapses the agents into one LLM whose single output is claim-checked.
Why not just…?
Why not just use SASRec and avoid LLMs entirely? This is a real threat. The masked condition is designed to test it. If agentic systems lose their advantage over SASRec once titles stop matching memory, then much of their reported benefit came from contamination, as the MovieLens memorization evidence suggests (Do LLMs Memorize). We would report that as a primary finding. The case for LLM agents then rests on explanations and constraint-following in the conversational variant, which SASRec cannot provide.
Why not just constrain outputs to valid catalog IDs? ToolRec's directory matching removes nonexistent items (Let Me Do It For You). It cannot catch a real item described with a remembered but wrong attribute, or a real item ranked high because of a fabricated reason. The conflict condition measures exactly these errors. We expect ID filtering to score near-perfect on item validity and no better than ARAG on claim precision.
Why not just check the final answer once instead of every handoff? If final-only checking matches per-handoff checking on ranking and constraint metrics, then cascades are not the issue. In that case the single-agent-plus-checker ablation is the honest recommendation and multi-agent decomposition is unnecessary. The per-agent error logs let us test the cascade mechanism directly. Errors introduced by the user-understanding agent should propagate into ranking decisions under final-only checking but not under per-handoff checking.
Why not just use an LLM judge such as MAIN-RAG or MACRec's Reflector? These judges are flexible and need no structured schema, which matters for attributes absent from the database. Their weakness is that they share the generator's memory. When record and memory disagree, they may side with memory. The judge ablation runs on the conflict condition to test exactly this. We predict the judge does well on the original condition and degrades on conflict items, and the lookup does not.
Proposed Experiments and Decisive Checks
Metrics cover three areas. Ranking quality is NDCG@5, Hit@5 and Recall@10 overall and on the least-popular 20% of items, a stratification motivated by the memorization study's popularity gap. Faithfulness covers three rates: claim precision against records in messages and explanations, the record-following rate on conflict items, and constraint-violation rate in the conversational variant. Cost is tokens per query and wall-clock latency, because the checker adds structured output but no extra LLM calls.
Three outcomes would be decisive. The first is the gap between the original and masked conditions for each LLM system relative to SASRec, which quantifies memorization. The second is whether claim-checked handoffs raise record-following on conflict items while keeping original-condition NDCG within one standard error of ARAG. The third is whether per-handoff checking beats final-only checking on conflict-item ranking. A null third result would mean the value lies in the checker rather than in decomposition, and we would state that plainly.
References
- Let Me Do It For You: Towards LLM Empowered Recommendation via Tool Learning
- Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1M
- Data-Driven Discovery of Dynamical Systems in Pharmacology using Large Language Models
- Adaptive Memory Admission Control For LLM Agents
- MACRec: A Multi-Agent Collaboration Framework for Recommendation
- ARAG: Agentic Retrieval Augmented Generation for Personalized Recommendation
- Do Language Models Know When They're Hallucinating References?
- MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation
- Logit Space Constrained Fine-Tuning for Mitigating Hallucinations in LLM-Based Recommender Systems
- Retrieval-Augmented Purifier for Robust LLM-Empowered Recommendation
- Cross-Scale Collaboration between LLMs and Lightweight Sequential Recommenders with Domain-Specific Latent Reasoning