A recommender that outputs a short list of items it is confident about, with a stated chance that the user's next choice is on that list, would be far more useful than one that outputs a ranked guess. Conformal prediction can build such lists. It is a post-hoc method that turns any model's scores into a set guaranteed to contain the true answer with probability at least $1-\alpha$, provided past and future data are exchangeable, meaning they behave like a shuffled deck. Recommendation breaks this premise in two ways. Tastes drift, so old interactions mislead. Data is extremely sparse, so a single user offers almost nothing to calibrate on. The standard fix for drift, down-weighting old data, makes the sparsity problem worse: discard the past and a user's calibration sample shrinks to a handful of points, and the sets swell toward the whole catalog.
We propose to resolve this trade-off by pooling calibration data across users who belong to the same latent taste group. These groups are inferred from each user's own interaction history and are allowed to change over time. A calibration point counts toward a test user's set according to two things: how recent it is, and how similar its user's group membership was to the test user's current membership. Users who drift in opposite directions are kept apart, and users who drift together share evidence. A single global calibration averages these groups into one threshold and so hides the users who are drifting away. We expect pooling within groups to give coverage that holds within each group under drift, with sets small enough to show a person.
Core Claim
The claim has two parts. The first is that in sparse, drifting recommendation logs, recency-only weighting of calibration scores fails in two specific ways. Coverage is marginally adequate but systematically low for groups of users whose tastes move against the majority. The effective calibration sample size is also too small to keep sets compact. The second is that weighting by recency multiplied by similarity of latent group membership fixes both. For a test user $u$ at time $t$, calibration score $s_i$ from user $u_i$ at time $t_i$ receives weight
$$w_i(u,t) = \gamma^{\,t - t_i}\,\kappa_\tau\big(m_u(t),\, m_{u_i}(t_i)\big),$$
where $\gamma \in (0,1]$ is a recency discount, $m_u(t)$ is user $u$'s soft membership vector over $K$ latent groups computed only from interactions before $t$, and $\kappa_\tau$ is a similarity kernel with bandwidth $\tau$. The prediction set contains every item whose score falls below the weighted $(1-\alpha)$ quantile of the $s_i$. The quantity that governs set size is the effective sample size
$$n_{\text{eff}} = \frac{(\sum_i w_i)^2}{\sum_i w_i^2},$$
which recency-only weighting collapses and group pooling restores.
Three lines of prior work constrain or motivate this design. Conformal prediction beyond exchangeability supplies the method. It shows that weighted conformal quantiles lose coverage only by a weighted sum of total-variation distances between score distributions, not between raw data distributions. That is exactly why matching calibration points to the test user's group should help: within-group score distributions are closer. That paper picks its weights in advance, leaves data-driven weight choice open, and works on regression. Non-Exchangeable Conformal Risk Control extends the same idea to general losses with a maximum-entropy weight heuristic, but its weights are also recency-only. Adaptive Conformal Prediction via Mixture-of-Experts Gating Similarity supplies our second ingredient. It weights calibration residuals by the similarity of mixture-of-experts gating vectors, and uses a randomization step on the test point's gating vector to keep marginal validity. Our membership kernel borrows that construction. That paper assumes stationary data and continuous regression; we add drift and a large discrete catalog.
Confidence-aware Fine-tuning of Sequential Recommendation Systems via Conformal Prediction motivates the problem directly. It shows conformal sets on sequential recommenders can shrink from about 300 to about 25 items through training-time conformal losses, and its authors name non-exchangeable calibration for temporal dependence as open future work. The heterogeneity side is motivated by Position: Aggregate Preference Optimization Hides a Posterior Identifiability Failure for Pluralistic Alignment. It shows that one model fit to polarized groups attains aggregate mean squared error of 0.008 but 0.06–0.07 per subgroup. A marginal conformal wrapper around such a model will therefore meet its target overall while missing it for groups. That paper also says identifiability returns once individual interaction traces are available, and recommendation logs have those traces.
Research Program
The first component is the latent group model. We will not cluster users by their taste embeddings. Backbones regularized like SimGCL push embeddings toward a uniform spread on the sphere, which works directly against finding clusters. Instead, following Class-Conditional Conformal Prediction with Many Classes, which clusters classes by the quantiles of their conformity scores, we describe each user by how the backbone errs on them. The description has two parts: recent score quantiles, and an attribute vector of recent interactions. A $K$-component mixture over these descriptions gives soft memberships $m_u(t)$ that are recomputed causally as history accumulates. The mixture form is the finite latent mixture standard in the choice-modeling literature, with recovery guarantees of the kind studied in Learning Mixed Multinomial Logits with Provable Guarantees. Users with too few interactions fall back to a global "null group," as Clustered CP does for rare classes.
The second component is calibration, which follows the weighted quantile rule above. We randomize the test membership vector as in MoE-CP so that marginal validity holds for the stationary part of the problem. We inherit the total-variation gap of the non-exchangeable bound for the drifting part, and we report that gap empirically rather than claiming it vanishes.
The third component addresses the open question both non-exchangeable papers leave unresolved: how to choose $(\gamma,\tau)$. We pick them on the most recent closed time window. The rule chooses the pair that minimizes average set size subject to empirical group-conditional coverage of at least $1-\alpha$ on that window, and then applies it to the next window. This rule is a heuristic and comes with no guarantee. We will measure how often it fails by comparing the coverage it predicts with the coverage it achieves.
The fourth component is analysis. We will trace $n_{\text{eff}}$, set size, and per-group coverage as functions of $\gamma$ and $\tau$. The aim is to show where pooling buys back the sample size that recency discounting removes, and where it breaks down because pooled users drift differently enough to raise the total-variation term.
Baselines
The naive reference point is split conformal prediction with uniform weights on a pretrained SASRec, calibrated on a held-out recent slice. This is the "do nothing beyond standard conformal" option. Standard conformal is known to fail under drift: coverage fell to 74.3% under a biased split in Conformal prediction beyond exchangeability, and to about 81% against a 90% target under posterior drift in Conformal Prediction Under Generalized Covariate Shift with Posterior Drift. Neither figure comes from recommendation data. We beat this baseline if we close its coverage shortfall on chronological test windows.
The strongest existing competitor is CPFT, the method of Confidence-aware Fine-tuning of Sequential Recommendation Systems via Conformal Prediction, with recency-weighted calibration from Conformal prediction beyond exchangeability added on top. This is what a careful practitioner would build today. It is the most efficient conformal recommender in the file, with sets reduced from about 300 to about 25 items at valid coverage on the Amazon subsets, and the add-on handles drift. No reported result was found on Lacuna for CPFT, or any conformal recommender, under a chronological split with drift-aware weights. We will therefore produce this number ourselves. Every method uses the same CPFT-fine-tuned backbone, the same calibration data, and the same grid budget for $\gamma$, selected by the same last-window rule. We beat it if our worst-group coverage is within 2 points of target where it is not, with average set size no more than 15% larger.
Two ablations isolate the key ingredient. The first is a hard-cluster version: one-hot memberships from k-means on the same descriptions, multiplied by recency. The second is membership weighting without recency, which gives an exchangeable Clustered-CP analogue over users. The backbone's own accuracy is reported for context only. On MovieLens-1m, A Systematic Replicability and Comparative Study of BSARec and SASRec for Sequential Recommendation reports BSARec NDCG@10 of 0.07704, and on Foursquare-NYC NDCG@5 of 0.2668 versus 0.2334 for SASRec. Our backbones should land near these values so that set sizes are not inflated by a weak model.
Why not just…?
Why not just use recency weighting with one global threshold? This is the real threat when drift is shared by everyone, and in that case we expect it to tie us. It fails when groups drift in different directions. A global weighted quantile then mixes score distributions that are moving apart, and the bound in Conformal prediction beyond exchangeability loosens by exactly that mismatch. The planted opposite-drift experiment below separates these cases. If global recency weighting matches us on worst-group coverage there, the group term is not earning its place, and we will say so.
Why not just calibrate each user separately with time decay? A typical user has a few dozen interactions, and the discounted effective sample size will often be in single digits. Under those conditions the conformal quantile is the maximum score or undefined, so the set is the whole catalog. Class-Conditional Conformal Prediction with Many Classes documents the same collapse for classwise calibration with 20–75 examples per class. We include per-user calibration as a reference curve, expecting it to dominate in coverage and be useless in set size.
Why not just run hard-clustered conformal with a decay? This is the closest simpler variant, and it could plausibly do as well. Our method contains it as the one-hot limit $\tau \to 0$. Soft membership should matter for users near group boundaries and for users moving between groups, whose hard assignment flips and resets their pool. If the hard version matches ours on worst-group coverage and set size, the contribution narrows to two things: the score-profile descriptions that avoid the uniformity problem, and the window-based weight selection. We will report it on those terms.
Why not just adapt the backbone faster, for example with discounted training as in NS-DPO? Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift keeps reward accuracy at 60–70% after preference flips where DPO falls to 50–55%. A better-tracking model does shrink the residual drift. However, it still fits one population, so it inherits the masking failure of the Position paper, and it produces no coverage statement. We test whether conformal weighting adds value on top of a discount-trained backbone, not instead of one.
Proposed Experiments
All experiments use strict chronological splits. Calibration always precedes test, and windows roll forward. The data are the five Amazon Review subsets used by CPFT (Scientific, Pantry, Instruments, Arts, Office), MovieLens-1m, and Foursquare-NYC. The backbones are SASRec and BERT4Rec fine-tuned with CPFT, plus a LightGCN-with-SimGCL model on Amazon-Book and Yelp2018 to test a backbone with uniform embeddings. The set must contain the user's next item at a 90% target.
We report five metrics. The first is marginal coverage per window. The second is worst-group and average group coverage gap, the CovGap measure of Clustered CP applied to inferred groups. Because inferred groups could flatter our own method, we also compute the gap over held-out groupings defined independently: user activity deciles and item-popularity tiers of the target item. The remaining metrics are average set size, $n_{\text{eff}}$, and the backbone's Recall@50 and NDCG.
The decisive check is semi-synthetic, so that the ground truth is known. On MovieLens-1m we plant two latent groups by genre affinity. Midway through the timeline, we flip one group's affinity and leave the other unchanged, reproducing the polarized-and-drifting regime of the Position paper inside real sparse logs. We predict three outcomes. Global recency weighting will hold near 90% marginally but under-cover the flipping group by several points. Per-user calibration will cover both groups with near-catalog sets. Our method will cover both groups within 2 points with sets comparable to global weighting.
A second check sweeps sparsity by subsampling interactions. It tests whether the advantage over global weighting grows as data thins, which is the regime where pooling should matter most.
A third check asks whether the window-based selection of $(\gamma,\tau)$ predicts next-window coverage. We track how often realized coverage falls below the value chosen on the previous window.
If the planted-flip result fails, meaning group weighting does not recover worst-group coverage beyond global recency weighting, the central claim is wrong. That outcome would indicate that score-profile groups do not track the direction of drift.
References
- Conformal prediction beyond exchangeability
- Non-Exchangeable Conformal Risk Control
- Adaptive Conformal Prediction via Mixture-of-Experts Gating Similarity
- Confidence-aware Fine-tuning of Sequential Recommendation Systems via Conformal Prediction
- Position: Aggregate Preference Optimization Hides a Posterior Identifiability Failure for Pluralistic Alignment
- Are Graph Augmentations Necessary?: Simple Graph Contrastive Learning for Recommendation
- Class-Conditional Conformal Prediction with Many Classes
- Learning Mixed Multinomial Logits with Provable Guarantees
- Conformal Prediction Under Generalized Covariate Shift with Posterior Drift
- A Systematic Replicability and Comparative Study of BSARec and SASRec for Sequential Recommendation
- Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift