Recommender systems tend to over-suggest a few popular items and bury the long tail. Different model families do this for different reasons. Matrix factorization lets the vectors of popular items grow long, so their scores dominate. Graph models such as LightGCN pass messages through high-degree "hub" items and amplify them. Today's fixes rewrite the training of one model at a time. This proposal moves the correction somewhere else: into a small second-stage model that blends the scores of several already-trained recommenders. That blender, a stacking meta-learner, has only a handful of weights. Those weights are constrained to be non-negative and sum to one, which is convex weight regularization. Because the blender is so small, we can train it with an aggressive popularity correction: reweighting each held-out interaction by the inverse of how likely its item was to be seen. That same correction is notoriously unstable when applied to models with millions of parameters.
The two-sentence version is this. Stacking fit on ordinary held-out interactions will faithfully reproduce popularity bias, because the held-out data are just as biased as the training data. If instead the stacker is a convex mixture whose weights may shift with item popularity, and it is trained on a popularity-reweighted held-out objective, the correction is cheap and its variance is controlled. Base models can then specialize, and the combination layer decides how much each one is trusted for head items versus tail items.
Core Claim
The obvious combination, stacking several recommenders with simplex weights, is not a debiasing method on its own, and we think this is the crux the literature has missed. Classical stacking (Stacked Generalizations: When Does It Work?, which fits linear meta-learners on out-of-fold class probabilities) optimizes accuracy on held-out data drawn from the same distribution as training. In recommendation, that distribution is the biased exposure log. A stacker fit to it will reward whichever base model best predicts popular-item clicks, which is usually the most popularity-biased one.
Our claim has three parts. First, the debiasing must live in the meta-objective. Second, the convex constraint is what makes a high-variance correction affordable there. Third, letting the weights depend smoothly on item popularity allows different base models to own different popularity tiers.
Concretely, let $s_k(u,\cdot)$ be base model $k$'s out-of-fold score for user $u$, converted into a probability distribution over items by a per-model softmax with a fitted temperature. Following Ting and Witten, we combine probabilities rather than raw scores. The stacked distribution is $p_w(u,i)=\sum_{k=1}^K w_k(n_i)\,s_k(u,i)$, where $n_i$ is item $i$'s training interaction count. The weight vector $w(n_i)=\mathrm{softmax}(A\,\phi(\log n_i))$ always lies on the simplex. Here $\phi$ is a small basis of $B$ smooth functions, so $A$ has only $K\times B$ entries. The meta-learner minimizes
$$\hat{R}(A)=\frac{1}{|\mathcal{H}|}\sum_{(u,i)\in\mathcal{H}} \frac{-\log p_w(u,i)}{\hat\rho_i} \;+\; \lambda\,\mathbb{E}_{i}\,D\big(w(n_i)\,\|\,\nu\big).$$
In this objective, $\mathcal{H}$ is the set of held-out interactions and $\hat\rho_i\propto n_i^{\eta}$ is a clipped, self-normalized popularity propensity. $D(\cdot\|\nu)$ is the KL divergence from a uniform prior $\nu$, and $\lambda$ sets its strength. With $B=1$ (static weights), the objective is convex in $w$. Because $-\log$ is convex, Jensen's inequality guarantees the mixture's loss is no worse than the weighted average of the base models' losses, the property Putting It All Together: Methods for Combining Neural Networks emphasized for convex ensembles.
The stability argument borrows from Data-Dependent Bounds for Bayesian Mixture Methods. That paper bounds the generalization gap of KL-regularized mixtures by a term in $\sqrt{(D(q\|\nu)+\log(1/\delta))/n}$, independent of how complex the base hypotheses are. We expect, heuristically, that importance weights inflate this term roughly by the square root of the largest clipped weight, since that factor rescales the bounded loss. The penalty stays tiny because the complexity is carried by $D$ and $K\times B$ parameters, not by embedding tables. We will test this expectation empirically rather than claim it as a theorem.
Relation to Prior Work
The closest work is Multiple Robust Learning for Recommendation. It combines several candidate propensity models and error-imputation models through a ridge-penalized least-squares fit, yielding an unbiased loss estimator for training one rating model on missing-not-at-random data. We borrow its lesson that combinations under small propensities need stabilization. Our setup differs in three ways. We combine the recommenders' outputs rather than auxiliary nuisance estimators. We target top-$N$ ranking under item popularity rather than rating error. And our weights vary with popularity rather than being global. The same paper also warns that heavy regularization reintroduces bias, which is why we sweep $\lambda$ and the clipping level instead of fixing them.
On the ensembling side, Unbiased, Effective, and Efficient Distillation from Heterogeneous Models for Recommender Systems motivates the proposal. It shows that MF, LightGCN and autoencoders are complementary, and it distills them into one student (HetComp). It never learns how to weight them and never evaluates popularity debiasing.
Ensemble Methods for Causal Effects in Panel Data Settings supplies the simplex-stacking template. It fits a cross-validated convex mix of vertical regression, horizontal regression and matrix completion. It also constrains our design: its fitted weights were often near 0 or 1. That pattern suggests plain model selection, possibly per popularity tier, might capture most of the benefit, and we test exactly that.
Ting and Witten likewise constrain us. They found non-negativity mainly helped interpretability, not accuracy. Our position is that under biased held-out data, the constraint earns its keep through variance control of the reweighted objective, not through raw accuracy, and that is the comparison we run. On different ensembles of kernel machines supports the variance story: its unconstrained combination weights overfit badly while the convex versions did not.
The single-model debiasers serve as base learners and as competitors. These are MACR (Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System, which subtracts a counterfactual estimate of the item-popularity effect at inference), Adap-τ (Adap-τ : Adaptively Modulating Embedding Magnitude for Recommendation, which normalizes embeddings and adapts the softmax temperature), and PAAC (Popularity-Aware Alignment and Contrast for Mitigating Popularity Bias, which aligns tail-item embeddings with co-interacted head items and reweights contrastive losses). PAAC's weaker gains on sparse Gowalla motivate our hypothesis that no single inductive bias suffices at every popularity level.
Research Program
We build a base pool of six recommenders trained on identical splits:
- BPR matrix factorization
- LightGCN (LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation)
- an autoencoder recommender as in HetComp
- MACR-LightGCN
- Adap-τ-MF
- PAAC-LightGCN, using the released code at
miaomiao-cai2/KDD2024-PAAC
Out-of-fold scores come from three-fold splitting of training interactions. Because six models times three folds is expensive, we also test a cheaper blending variant: train base models once and fit the meta-learner on a single held-out slice. We report whether the two give the same weights.
The convex hull limitation is real. If every base model over-ranks head items, no convex mixture escapes. We confront it in two ways. The pool deliberately includes the debiased models. We also run a hull diagnostic: fit the stacker directly on the uniform test distribution, which is an oracle, to measure how much headroom a convex mixture can possibly offer.
The analysis has three parts. We plot the learned weight curves $w(n_i)$ to see which model owns which popularity tier. We measure variance of the learned weights across random seeds and resampled held-out sets as a function of clipping level and $\lambda$. And we compare the $B=1$ static mixture against the popularity-conditioned one. For deployment cost we optionally distill the final stacker into one student with HetComp-style training. This is secondary; the scientific question is whether the combination layer is the right place to debias.
Baselines
The naive reference point is the best single non-debiased base model, LightGCN or MF chosen by validation, scored on the same unbiased test sets. Next to it is uniform averaging of the same six calibrated probability outputs. Uniform averaging is a convex mixture that needs no fitting, so beating it isolates the value of learned weights.
The strongest existing single-model competitor is PAAC on LightGCN. On Yelp2018, Cai et al. report gains of 30.2% in NDCG@20 and 8.7% in Recall@20 over their baselines, and an 87.5% reduction in the popular-versus-unpopular accuracy gap. The gap reduction on Gowalla is 34.18%. All of these are relative figures from Popularity-Aware Alignment and Contrast for Mitigating Popularity Bias. For MACR, Wei et al. report average HR@20 improvements of 153% on MF and 241% on LightGCN across five datasets. For the multiple-robust estimator on Coat and Yahoo! R3, no reported result was found on Lacuna for absolute MSE, AUC or NDCG values. We therefore rerun PAAC, MACR and MR ourselves, with each method receiving the same hyperparameter search budget (same number of trials and same GPU hours) and selection on the same validation split. "Beating PAAC" means higher NDCG@20 on the unbiased test split, with a tail-tier NDCG@20 no worse than PAAC's, averaged over five seeds with non-overlapping confidence intervals.
The strongest ensemble competitor is unweighted stacking: the same convex popularity-conditioned meta-learner fit without propensity weights. This is the ablation that isolates our key ingredient. If it matches the full method on unbiased tests, the reweighted objective is not doing the work.
A second ablation removes the simplex: an unconstrained linear meta-learner, and a small MLP meta-learner, both fit with the same reweighted objective. If our variance story is right, these should show larger seed-to-seed weight variance and worse tail metrics.
A third ablation sets $B=1$, giving static weights, to test whether popularity conditioning matters.
HetComp distillation of the uniform ensemble is the efficiency reference. Its reported Foursquare figures are Recall@10 of 0.1263 for the student versus 0.1265 for the full ensemble, with 2.12s versus 35.47s inference time, in Unbiased, Effective, and Efficient Distillation from Heterogeneous Models for Recommender Systems.
Why not just…?
Why not just subtract a tuned popularity penalty from the best model's scores? Re-ranking by score minus $\alpha\log n_i$ is the bluntest and most likely threat. It is one parameter, cheap, and often strong. We treat it as a first-class baseline, applied to every base model with $\alpha$ tuned on the same budget. We also note that if the pool includes a pure popularity scorer, our popularity-conditioned stacker can represent a comparable tier-dependent shift. If the tuned penalty on the best model matches us on the uniform test sets in both head and tail tiers, our claim fails. We would then report that heterogeneity adds nothing beyond a scalar correction.
Why not just average the models uniformly? Uniform averaging is free and automatically convex, and Ting and Witten found that constraints mainly bought interpretability. But averaging gives equal say to the most biased models. The panel-data stacking of Athey et al. found near-0/1 optimal weights, far from uniform. Yamana et al.'s convex weights beat bagging, for example 21.6% versus 22.8% error on Diabetes. Our headline comparison is against uniform averaging, both overall and per tier.
Why not just pick the best model per popularity tier, or tune mixture weights on a small unbiased validation slice? Athey et al.'s near-corner weights make tier-wise selection plausible, so we include it, using held-out reweighted loss to choose the model per tier. Tuning on uniformly collected data is the oracle-flavored alternative that Coat and Yahoo! R3 permit, and we run it. The difference is that our method needs no randomized data, which most platforms lack. The question is how much of the oracle's gain it recovers.
Why not just use the best single debiaser, such as PAAC? If one debiaser dominates at every tier, the learned weights will collapse onto it, and the method reduces to model selection with no gain. The weight-curve plots and the tail-tier comparison against PAAC on Gowalla, where PAAC's gap reduction was weakest, settle this directly.
Proposed Experiments
Datasets come from the file's debiasing protocols. Coat and Yahoo! R3 have randomly collected test sets. Yelp2018, Gowalla and Amazon-book use the uniform-item test splits of the PAAC and Adap-τ evaluations. Semi-synthetic MovieLens 100K with three increasing bias levels follows Li et al. Metrics are NDCG@20 and Recall@20 overall and split by item popularity tier, the head–tail accuracy gap, and seed-to-seed variance of the learned weights.
The first decisive check is the hull diagnostic. If the oracle convex mixture fit on uniform test data does not beat the best single debiased model, there is no headroom, and the project stops at a negative result.
The second check is the fraction of that oracle headroom recovered by the reweighted stacker versus the unweighted stacker. Our hypothesis predicts a large recovery for the former and near zero for the latter.
The third check uses MovieLens 100K. As the synthetic bias level rises, the unweighted stacker and unconstrained meta-learners should degrade, Li et al. report competitor drops of up to 36.7%, while the convex reweighted stacker should stay flat, and its tail-tier weight should shift toward the debiased base models.
The fourth check is the head-to-head against the tuned popularity penalty and PAAC under matched tuning budgets. This one decides whether the combination layer is a better place to debias, or merely an equally good one.
References
- Stacked Generalizations: When Does It Work?
- Putting It All Together: Methods for Combining Neural Networks
- Data-Dependent Bounds for Bayesian Mixture Methods
- Multiple Robust Learning for Recommendation
- Unbiased, Effective, and Efficient Distillation from Heterogeneous Models for Recommender Systems
- Ensemble Methods for Causal Effects in Panel Data Settings
- On different ensembles of kernel machines
- Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System
- Adap-τ : Adaptively Modulating Embedding Magnitude for Recommendation
- Popularity-Aware Alignment and Contrast for Mitigating Popularity Bias
- LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation