Balancing Rank Regularization Item by Item in Recommenders

Recommender systems learn a vector for every user and item. Left alone, these vectors tend to crowd into a few directions of the available space, a failure called dimensional collapse. The crowding hurts rarely-seen "long-tail" items most, which is one mechanism behind popularity bias: the system keeps surfacing hits because niche items no longer have room to be told apart. A clean fix that needs no negative samples already exists. You add a regularizer that keeps the item vectors' covariance full-rank, such as a log-determinant penalty or a Barlow Twins–style decorrelation term. The trouble is that every paper using such a regularizer reports that results hinge on one weight $\lambda$ trading the ranking loss against the regularizer.

Our claim is that this sensitivity is not an ordinary tuning nuisance. It is a structural mismatch. A popular item receives ranking-loss gradients orders of magnitude larger than a tail item, while the regularizer pushes on every item in the batch with roughly comparable force. So any single $\lambda$ leaves head items dominated by the ranking loss and tail items dominated by the regularizer, and no global value is right for both groups. We propose to set the balance separately for each item's vector, using the ratio of its own running gradient norms from the two losses. The rule is cheap because in ID-based recommenders each item's vector is its own bottleneck. It makes one testable prediction: if per-item balancing is the right fix, the best fixed weight for each popularity bucket, found by brute-force search, should track that bucket's gradient-norm ratio.

Core Claim and Mechanism

Write $e_i$ for item $i$'s final representation, meaning the raw embedding in matrix factorization or the propagated output in LightGCN. Write $\bar g^{\text{rec}}_i$ and $\bar g^{\text{geom}}_i$ for exponential moving averages of the norms of $\nabla_{e_i}\mathcal{L}_{\text{rec}}$ (the ranking loss, BPR or cross-entropy) and $\nabla_{e_i}\mathcal{L}_{\text{geom}}$ (a negative-free rank regularizer). The update applied to item $i$'s representation is

$$ g_i = \nabla_{e_i}\mathcal{L}_{\text{rec}} + \rho\,\frac{\bar g^{\text{rec}}_i}{\bar g^{\text{geom}}_i+\epsilon}\,\nabla_{e_i}\mathcal{L}_{\text{geom}}, $$

where $\rho$ is a single dimensionless ratio. It replaces $\lambda$ and has the same meaning for every item: how hard geometry pushes relative to that item's own task signal. The combined row gradients are then sent back through the rest of the model in one backward pass.

This borrows the key move of MultiBalance. That paper balances objectives at a small shared representation layer to avoid one backward pass per loss over billions of embedding parameters, which it reports costs other balancing methods a 70–80% throughput loss. MultiBalance does this across prediction tasks such as click-through and conversion, and it computes one global weight vector. We apply the same trick between a task loss and a geometric regularizer, at the granularity of individual item rows, so that the balance can differ between head and tail.

A second, smaller ingredient concerns which items the regularizer looks at. Batch covariance estimated only from in-batch positives is itself popularity-skewed, because popular items are sampled as positives far more often. We therefore estimate $\mathcal{L}_{\text{geom}}$ on a uniformly drawn set of item rows. These rows are never scored against users and are not negatives in the contrastive sense, so the method stays negative-free.

Relation to Prior Work

Three papers motivate the gap by showing that rank preservation helps the tail while leaving the trade-off weight static.

Mitigating the Popularity Bias of Graph Collaborative Filtering: A Dimensional Collapse Perspective proves that graph convolution shrinks singular values and crushes tail items into degenerate subspaces. It counters this with a LogDet barrier (GCF$_{logdet}$), with a single grid-searched $\lambda$.

Barlow Twins for Sequential Recommendation applies redundancy reduction to SASRec without negatives, which raises Gowalla item coverage by 25.9–36.3%. It explicitly reports high sensitivity to its weight $\alpha$ and to interaction skew. We read that skew sensitivity as a direct symptom of the head/tail mismatch described above.

Balancing Embedding Spectrum for Recommendation optimizes the embedding spectrum directly (DirectSpec) and shows that Barlow Twins is a special case. It never measures popularity-stratified effects. We reuse its regularizer as one choice of $\mathcal{L}_{\text{geom}}$, alongside LogDet and a VICReg-style variance-plus-covariance term.

Two lines of work constrain the design. Adap-τ shows that normalizing vector length removes magnitude-driven popularity bias, and it derives temperature in closed form. However, the GCF$_{logdet}$ paper shows that unit-norm vectors can still lie on a low-rank disc, so normalization alone is not a substitute for rank control. We keep unit normalization with Adap-τ's temperature as part of the backbone, so that our gains cannot be explained by magnitude effects.

Mitigating Popularity Bias … A Gradient Perspective shows that unbalanced positive and negative gradients rotate items toward a single popular direction, and it subtracts that direction after training with two hand-tuned scales. Subtracting one direction cannot restore dimensions that collapsed during training. That paper's gradient-imbalance diagnosis is, however, exactly what makes per-item balancing plausible.

PAAC aligns tail items to popular co-occurring anchors and reweights contrastive pairs. It is a strong competitor, but it keeps negative sampling and adds several static weights of its own.

Finally, general-purpose balancing methods weight entire losses with one global scalar or project out conflicts over the full parameter vector. GradNorm and CAGrad are examples. Neither can express "regularize the tail differently from the head."

Research Program

The work proceeds in three stages, and the first can kill the idea.

The first stage is diagnostic. On LightGCN and MF with the BPR loss, we bucket items into five popularity quantiles. For each bucket we grid-search a separate fixed weight $\lambda_b$, holding the other buckets at the global optimum, and we record the median gradient-norm ratio $\bar g^{\text{rec}}/\bar g^{\text{geom}}$ per bucket. The prediction is that the oracle $\lambda^*_b$ varies monotonically with popularity and is roughly proportional to the bucket's gradient ratio. If $\lambda^*_b$ is flat across buckets on most datasets, the per-item premise is wrong. In that case we would report that negative result and fall back to studying global balancing.

The second stage builds the per-item rule above for three regularizers (LogDet, DirectSpec, and the VICReg-style term) and for two populations used to estimate the regularizer (in-batch positives versus uniform item rows). We measure accuracy, tail accuracy, effective rank of the item matrix (the exponential of the entropy of its normalized singular values), and the training-time overhead of the extra gradient pass to the representation rows.

The third stage transfers the rule to sequential recommendation. There we replace BT-SR's $\alpha$ with the same per-item rule applied to the target-item vectors of SASRec.

Baselines

The naive reference point is LightGCN, and MF trained with BPR, with no rank regularizer. This tells us how much of any gain comes from regularization at all.

The strongest geometric competitor is GCF$_{logdet}$. On Yelp2018 its paper reports Recall@20 of 0.0725 against LightGCN's 0.0639, with tail-item improvement 1.51 times the baseline's. We rerun it on the same splits with a $\lambda$ grid of the same size as our grid over $\rho$, so both methods get equal tuning budgets. It counts as beaten only if we match its overall Recall@20 while improving tail-bucket Recall@20 and NDCG@20, and only if we do so without any per-dataset search over $\rho$.

The strongest debiasing competitor is PAAC. Its paper reports a 30.2% NDCG@20 gain over the strongest baseline on the unbiased Yelp2018 split and an 87.5% reduction in the head–tail accuracy gap. The file gives no absolute PAAC values, so no reported result was found on Lacuna for absolute PAAC Recall@20 or NDCG@20, and we will reproduce it with its released settings. Beating it means lower or equal head–tail gap on the uniform test splits without using negatives in the regularizer.

We also include three other existing methods, with equal tuning budgets:

  • Adap-τ as the strongest normalization-only method.
  • DirectSpec+ as the strongest spectral method.
  • The post-hoc popular-direction subtraction method on MovieLens-10M. Its paper reports roughly 37% Recall@20 and 51% NDCG@20 gains over MF; no absolute values are on Lacuna.

Three ablations isolate the key ingredient:

  • Global gradient-ratio balancing. The same rule, but using whole-batch gradient norms instead of per-row norms. This separates "adaptive" from "per-item."
  • Oracle per-bucket fixed weights. The $\lambda^*_b$ from stage one, which serves as a ceiling on what any popularity-dependent weighting could achieve.
  • Regularizer population. In-batch versus uniform item rows for estimating $\mathcal{L}_{\text{geom}}$.

Why not just…?

Why not just grid-search $\lambda$ carefully? In academic settings this is cheap, so it is a real threat. A tuned global $\lambda$ cannot give head and tail different balances, which is why the stage-one oracle is the deciding experiment. If the global optimum matches the oracle per-bucket configuration, our claim fails. Beyond accuracy, we will report the width of the hyperparameter range that stays within 95% of best accuracy, for $\lambda$ versus $\rho$, together with cross-dataset transfer: using the $\rho$ chosen on Yelp2018 unchanged on Gowalla and Amazon-Book.

Why not just use GradNorm or PCGrad on the two losses? These assign one weight per loss, or resolve conflict over the whole parameter vector. Applied naively over embedding tables they also hit the throughput wall that MultiBalance documents. Our global-balancing ablation is effectively GradNorm restricted to the representation layer. If it matches the per-item rule, the per-item contribution is empty, and we will say so.

Why not just reweight the regularizer by a fixed function of popularity, such as $\lambda_i \propto n_i^{-\beta}$ where $n_i$ is item $i$'s interaction count? This is the most serious alternative. If stage one confirms degree dependence, a one-parameter power law might capture most of it. We will fit $\beta$ by the same budget and compare. If it ties, we adopt it and report the gradient-ratio view as the explanation for why that form works. The remaining question would then be whether $\beta$ transfers across datasets as well as $\rho$ does. The gradient-ratio rule has one advantage a fixed power law cannot: it tracks changes during training, as head items saturate and their task gradients shrink.

Why not just normalize embeddings and subtract the popular direction afterwards? Both Adap-τ and the subtraction method are run as baselines. The GCF$_{logdet}$ analysis predicts that neither restores lost rank. The effective-rank measurements will show directly whether their tail gains come with recovered dimensions or merely with re-scored head items.

Proposed Experiments and Decisive Checks

The graph collaborative filtering experiments use Yelp2018, Gowalla, and Amazon-Book with the unbiased uniform test splits used by PAAC. Debiased evaluation uses MovieLens-10M with the uniform split from the gradient-perspective paper. Sequential experiments use MovieLens-1M and Amazon-Beauty with SASRec and BT-SR, where we report HR@1. BT-SR's paper reports a 12.7% HR@1 gain on MovieLens-1M, with no absolute value on Lacuna. Every result is the mean of five seeds and is reported overall and per popularity bucket.

Four outcomes decide whether the idea holds:

  • Degree dependence. The oracle $\lambda^*_b$ must vary with popularity and correlate with the gradient-norm ratio across at least three of the four graph datasets.
  • Sensitivity. The per-item rule must widen the near-optimal hyperparameter range relative to static $\lambda$, and its $\rho$ must transfer across datasets without retuning.
  • Accuracy. It must close most of the gap to the oracle per-bucket configuration, improve tail metrics over a tuned GCF$_{logdet}$ and over the fixed power-law weighting at equal overall Recall@20, and raise effective rank.
  • Cost. The per-row gradient pass must cost a small fraction of an epoch, which we will measure in wall-clock time.

Failure on the first check ends the per-item thesis. Failure on the third check while passing the second would reduce the contribution to tuning robustness, which is still a useful result but a smaller one.