The Language Model Already Knows Who Would Click Anyway: Parametric Priors as Free Baselines for Uplift in LLM Recommenders

A recommender should show people things they would not have found on their own. Most systems instead learn to predict clicks, and many of those clicks are "convenience clicks" by users who were going to reach the item anyway. One observational study estimated that up to 75% of recommendation clicks are of this kind (Estimating the Causal Impact of Recommendation Systems from Observational Data). The causal fix is to rank by uplift, meaning the difference between a user's engagement probability when the item is shown and when it is withheld. Uplift is hard to estimate because nobody observes both outcomes for the same user and item. Methods that work usually need a slice of randomized exposure traffic, which is costly to collect.

This note proposes that large language model (LLM) recommenders already carry a usable estimate of the "withheld" outcome. We can read it off by treating the recommender's behavior as a knowledge conflict, meaning a clash between what the model memorized in pre-training (its parametric knowledge) and what the prompt tells it (its contextual knowledge, here the user's history). The specific bet is this. An LLM's score for an item with the user's history removed reflects global, memorized appeal, which is a plausible proxy for how likely the item is to be found organically. The extra score that the user's history adds reflects personal fit. Ranking by that added score rather than by the total should move recommendations away from users who would engage regardless ("Sure Things") and toward users whose engagement depends on the recommendation ("Persuadables"). Chain-of-thought prompting, which asks the model to write out intermediate reasoning before answering, enters as the main threat to this decomposition. We will test whether it makes the model lean harder on memorized popularity and rationalize Sure Things, and whether a reasoning format that explicitly asks about the no-exposure outcome reverses that.

Core Claim

The claim has three parts, each falsifiable. First, in an LLM recommender, the gap between the score with the user's history and the score without it is a better uplift ranking signal than the full score. It should also be competitive with causal recommenders that require randomized logs. Second, the parametric (history-free) score predicts organic engagement beyond what raw item popularity counts in the interaction log predict. The advantage should be largest for new or rarely logged items, where log counts carry no information but the model may still know the item from pre-training. Third, free-form chain-of-thought increases the share of the final score explained by the parametric prior. A counterfactual reasoning format, which asks "would this user reach this item without being shown it?", reduces that share.

The closest existing work is Uplift-RAG. It scores retrieved documents by the difference in answer quality with and without the document. It found that 45.2% of top-retrieved documents mislead Llama-3-8B while only 17.8% help. Uplift-RAG motivates our with/without contrast, but it measures the uplift of context on the model for factual question answering. We use the same contrast to estimate the uplift of a recommendation on a user, and we validate it against behavioral treatment effects rather than answer accuracy. That substitution is the whole bet, and it can fail.

Positioning Against Prior Work

Three groups of papers shape the design.

The causal recommendation literature supplies the target and the strongest competitors. Recommendations as Treatments frames recommendations as interventions. Causal Embeddings for Recommendation (CausE) jointly factorizes a large biased log and a small randomized log, tying the two item embeddings together with a penalty. Causal Intervention for Leveraging Popularity Bias in Recommendation (PDA) treats popularity as a confounder, removes it during training through backdoor adjustment, and reinjects it at inference. Estimating Propensity for Causality-based Recommendation without Exposure Data uses item popularity as a prior for exposure propensity when no randomized log exists. All of these operate on ID-based logs. None can use what a pretrained model knows about an item it has rarely logged, and that is the gap our second claim targets. The last of these papers is also the most dangerous comparison, because it already says "popularity is a good enough baseline."

The knowledge-conflict literature supplies the mechanism and a constraint. A Causal View of Entity Bias in (Large) Language Models shows that entities seen frequently in pre-training lead models to ignore prompt context. We read this as evidence that popularity bias in LLM recommenders is partly a parametric override, not only a log artifact. Causal Direct Preference Optimization shows that standard preference tuning of generative recommenders amplifies spurious co-occurrences. That paper constrains our design: we deliberately use frozen or lightly fine-tuned models so we are not measuring a bias introduced by tuning. Cutting Off the Head Ends the Conflict (PH3) identifies attention heads that pull from memory versus heads that pull from the prompt. ContextFocus steers context adherence with a single activation vector. Both supply single-pass alternatives we must beat or explain. Both were evaluated only on short factual question answering, so their behavior on ranking is unknown.

The chain-of-thought literature supplies the variable under test. CoT4Rec and Do Mentioned Items Truly Matter? use written reasoning to describe a user's taste or explain why an item matches. That reasoning is about preference matching, never about the no-exposure outcome. Chain-of-Note shows that structured notes classifying each piece of context improve robustness to noisy retrieval. It is the template for our counterfactual note format, and its 20x latency cost (0.6 to 12.0 seconds per sample) is a constraint we report against. Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoning perturbs individual reasoning steps to measure their causal effect on the answer. We reuse that perturbation idea to check whether the written counterfactual reasoning actually drives the ranking or merely accompanies it.

Research Program

The basic object is a decomposition. Let $s(u,i \mid h_u)$ be the LLM recommender's log-probability of recommending item $i$ to user $u$ given history $h_u$. Let $s(u,i \mid \tilde h)$ be the same score under a null history $\tilde h$. The contextual gain is

$$g(u,i) = s(u,i \mid h_u) - s(u,i \mid \tilde h),$$

and the parametric prior is $p(i) = s(u,i \mid \tilde h)$, averaged over null histories. The null history can be empty, which risks confounding the comparison with prompt-length effects. It can be a random other user's history, which keeps the prompt format but removes personal fit. Or it can be the user's own history with item identities shuffled. We use all three and report whether conclusions agree. Disagreement between them is itself a diagnostic of where the memorized prior lives.

The work has four stages.

Stage one is a measurement study. Across open models of two sizes, we measure how much of $s(u,i \mid h_u)$ is explained by $p(i)$. We also measure how strongly $p(i)$ correlates with log popularity and with pre-training familiarity, estimated by how accurately the model can describe the item when given only its title.

Stage two tests the Sure-Thing hypothesis against behavioral ground truth (described under experiments). The question is whether $p(i)$ predicts organic engagement, and whether ranking by $g$ rather than $s$ increases realized uplift.

Stage three introduces reasoning. We compare direct scoring, free-form chain-of-thought in the CoT4Rec style, and a counterfactual note. The note writes, per candidate, a short judgment of whether the user would plausibly reach the item without being shown it and whether the history gives specific evidence of fit, before scoring. For each format we recompute the parametric share of the final score. For the counterfactual note, we perturb the "would reach it anyway" sentence, following Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoning, to test whether the ranking actually depends on it.

Stage four is mechanistic and deliberately small. We ask whether the memory-type heads found by PH3 on factual conflicts are the same heads that carry $p(i)$ in recommendation prompts. We test this by ablating them and checking whether $p(i)$ collapses while $g$ survives. If it does, there is a single-pass route to the same ranking without written reasoning.

Baselines

The naive reference point is the LLM recommender ranked by its total score $s(u,i \mid h_u)$ with direct prompting. This is what current LLM recommenders do, and the proposal claims it systematically favors Sure Things. Beating it on realized uplift is necessary but far from sufficient.

The most dangerous baseline is popularity subtraction. We rank by the LLM's total score minus a fitted multiple of log item popularity, with the multiple tuned on the same validation split. We also run the full PDA method from Causal Intervention for Leveraging Popularity Bias in Recommendation, trained on the same logs. PDA reports up to a 119% improvement across Recall, Precision, Hit Rate, and NDCG on Kwai, though that was measured against biased baselines on standard accuracy metrics, not on uplift. This baseline is strongest because, if the model's parametric prior is mostly memorized popularity, subtracting popularity directly yields the same ranking more cheaply. Our method beats it only if ranking by $g$ yields higher uplift than popularity subtraction overall and the gap widens on items with few log interactions. A gain confined to popular items would mean we have rediscovered popularity adjustment.

The strongest causal baseline that uses randomized data is CausE (Causal Embeddings for Recommendation), trained with access to the randomized slice. On MovieLens10M it reports a 21% MSE lift and 20% NLL lift over the propensity-weighted WSP2V-blend, and roughly 5% AUC over BanditNet. We will rerun it under its SKEW protocol rather than quote those numbers as directly comparable. Our method uses no randomized data. Matching CausE within its confidence interval would therefore count as a success, and beating it is not required. We also report an IDUM-style invariant uplift network (Invariant Deep Uplift Modeling) where it can be trained. Its reported AUUC of 0.2847 on a proprietary short-video dataset is not reproducible here and serves only as context.

The reasoning baselines are CoT4Rec-style free-form preference reasoning and Chain-of-Note-style relevance notes without the counterfactual question. Each uses the same backbone, the same candidate sets, and the same token budget cap. The single-pass mechanistic baselines are ContextFocus steering toward the user history and PH3 memory-head pruning. Each is tuned on 1,500 recommendation prompts to match the data regime in which ContextFocus was reported to converge.

The key ablation replaces the parametric prior $p(i)$ with log popularity inside the same gain formula. If that ablation matches the full method, the language model contributes nothing beyond counting.

Why not just…?

Why not just subtract item popularity? This is the real threat, and the proposal is built so it can win. Estimating Propensity for Causality-based Recommendation without Exposure Data already shows popularity is an informative prior for exposure. If $p(i)$ correlates with log popularity above about 0.9 and the popularity-swap ablation matches our ranking, we will report that the LLM decomposition is redundant. In that case the remaining contribution is diagnostic: a measured explanation of why LLM recommenders favor Sure Things. The case for the model's prior rests on items whose organic appeal is not visible in the log, such as new releases of known franchises or items that are widely known but rarely logged on this platform. The cold-start split is the decisive check.

Why not just use a standard uplift model like CausE or a tabular learner? When a randomized slice exists and items are well logged, these are cheaper and may win, which Causal Embeddings for Recommendation and Invariant Deep Uplift Modeling both support. Our argument is about regime, not dominance: the decomposition needs no randomized traffic, and collecting that traffic is a known cost of the CausE family. The head-to-head with CausE, plus a curve showing how CausE degrades as its randomized slice shrinks toward zero, settles where each one wins.

Why not just ask the model in its chain-of-thought whether the user would click anyway? This is our counterfactual note arm, so it is tested rather than dismissed. The concern is that written reasoning can be a fluent rationalization disconnected from the score. Pure prompting also cannot tell us what the model's prior is; it can only tell us what the model says. If the note alone matches $g$ on uplift and survives the step-perturbation test (the ranking actually changes when the "would reach it anyway" sentence is altered), we will adopt it as the simpler method. If the ranking ignores the perturbation, the note is decorative, and the arithmetic decomposition is the honest signal.

Why not just steer the model toward the user's context with ContextFocus or PH3? Pushing the model to rely more on the history raises the contextual share of the score. But it does not separate an item that fits the user from an item the user would find anyway, because a popular item that also fits the history still scores high. ContextFocus was also validated only on single-hop factual question answering. We run both methods as baselines. If memory-head pruning reproduces the $g$ ranking in a single pass, that is a positive result for stage four and a cheaper deployment path, not a refutation.

Proposed Experiments

Behavioral ground truth is the hardest part, so we use three settings of decreasing realism and increasing control.

The first setting is KuaiRand, whose randomly exposed videos give engagement rates under exposure free of the platform's selection. We estimate an organic baseline from users' engagement with the same items in the standard log, restricted to interactions not preceded by a recommendation impression where the log allows this. We state plainly that this is an approximation of the withheld arm. Short-video items are weakly represented in LLM pre-training, so KuaiRand is a stress test where the parametric prior should help least.

The second setting is MovieLens10M and Netflix under the SKEW protocol of Causal Embeddings for Recommendation. Here items are films the model knows well, and CausE gives a directly comparable baseline on the same splits.

The third setting is a semi-synthetic benchmark built on MovieLens. Potential outcomes under exposure and non-exposure are generated from fitted user taste plus an organic-discovery term that we control. The organic term is set either to track log popularity, to track pre-training familiarity, or to be independent of both. The ground truth for Persuadables and Sure Things is therefore exact, and we can see when each baseline should win.

We also run ReDial (Do Mentioned Items Truly Matter?, which reports Recall@10 of 28.0% against 24.0% for CTA-CRS). There the conflict is explicit: a user criticizes a famous film, and the model's parametric fondness for that film competes with the stated dislike. On ReDial we measure how often each method still recommends the criticized item.

Metrics are area under the uplift curve and Qini on the real and semi-synthetic sets, and Recall@10 and NDCG on standard accuracy so readers can see what the causal ranking costs in raw engagement. We also report the parametric share of the score, defined as the variance in $s$ explained by $p$, for every prompting format, and tokens and seconds per user. On the CausE SKEW splits we additionally report MSE and NLL lift so the comparison is in that paper's own metrics. For the free-form chain-of-thought in the CoT4Rec style, no reported uplift result was found on Lacuna, and the same holds for any LLM recommender. Every uplift number for LLM methods will be new.

Three checks decide the outcome. If ranking by $g$ beats the naive total score on uplift but loses to popularity subtraction on the cold-start split, the second claim is false, and the project becomes a measurement paper about Sure-Thing bias in LLM recommenders. If free-form chain-of-thought does not raise the parametric share relative to direct scoring on any backbone, the third claim is false. If the semi-synthetic benchmark shows $g$ winning only when the organic term tracks pre-training familiarity, we will report that the method is only as good as the overlap between what the model read and what users find on their own. That is a narrower but still useful finding, and it tells practitioners which catalogs the method suits.