When engineers want to know what a recommender has learned, a common step is to cluster its item vectors and have a large language model (LLM) read a few items from each cluster and write a short name for it, such as "budget Thai takeout" or "1990s indie rock." This step quietly assumes that the items the LLM sees stand in fairly for the cluster. In recommendation data that assumption fails. A few popular items collect most of the interactions. Their vectors grow longer in proportion to how often they appear (Adap-τ). Popular and unpopular items also drift into separate regions of the vector space (PAAC). As a result, the clusters partly sort items by popularity rather than by meaning, the exemplars nearest each cluster center are mostly hits, and the labels end up describing the head of the catalog. The long-tail items that make up most of each cluster go undescribed.
We propose to measure this distortion and to correct it. The correction is a labeling procedure that removes the popularity signal from the geometry before clustering, chooses exemplars that represent tail members, and checks each label against the cluster's tail. We then use the corrected labels for a practical purpose: a semantic audit that names which concepts a recommender starves of exposure, and that tests whether popularity-debiasing methods actually fix those concepts or only shift aggregate metrics.
Core Claim
Our claim has two parts. First, standard LLM cluster labeling of recommender item embeddings is biased toward popular items in a way we can measure: a label recovers a cluster's head members much better than its tail members. Second, most of this gap comes from the clustering geometry, not only from which exemplars are chosen. That second part matters because it predicts that the cheap fix of sampling exemplars uniformly will not be enough. If it is enough, the method collapses to a sampling trick, and we will report that.
The metric is a tail-fidelity gap. Each label is turned into a text query and run against item metadata such as titles, categories, tags, or review text. We then compute recall of the cluster's members separately for head items (the top 20% by interaction count) and tail items:
$$\Delta_c = R_c^{\text{head}} - R_c^{\text{tail}},$$
where $R_c^{\text{head}}$ and $R_c^{\text{tail}}$ are the fractions of cluster $c$'s head and tail members retrieved in the top-$|c|$ results for the label query. An unbiased label should give $\Delta_c \approx 0$, and its tail recall should not fall below what the same retriever achieves with the cluster's ground-truth metadata categories.
Research Program
The project has four parts: a diagnosis, a labeling pipeline, a verification step, and an audit.
The diagnosis stage trains matrix factorization and LightGCN on Yelp2018, Amazon-book, MovieLens and Last.fm, which are the benchmarks shared by MACR, PAAC, Adap-τ and ICPE. We cluster the item embeddings with k-means and label each cluster with an LLM in the style of SynCID, which shows the LLM the items nearest the cluster center. We then report $\Delta_c$ for every cluster. We also measure how much of each cluster's membership is explained by popularity decile alone, using the normalized mutual information between cluster IDs and popularity deciles. This tests whether the separation that PAAC showed in t-SNE plots carries over to cluster assignments.
The labeling pipeline makes three changes, each meant to remove one source of head bias. The first change is to the geometry. We normalize item vectors, which removes the norm inflation described by Adap-τ. We then project out the single direction $w$ that best predicts log-popularity, fit by ridge regression on the normalized vectors:
$$\tilde v_i = \hat v_i - (w^\top \hat v_i)\,w, \qquad \hat v_i = v_i / \|v_i\|.$$
This is a geometric analogue of the way MACR subtracts the direct item-popularity effect from scores. Clustering is run on $\tilde v$. The second change is to exemplar choice. Exemplars are chosen to cover the cluster's spread in $\tilde v$, using farthest-point selection within popularity strata, rather than by closeness to the center. The third change is verification. The LLM proposes a label, the retrieval test computes tail recall, and the LLM is shown the tail members the label missed and asked to revise it. Only the retrieval score decides whether a revision is kept, never the LLM's own judgment. This addresses the hallucination and over-generalization risk that SynCID and Hanley et al. note for generative labeling.
Removing a popularity direction can also remove real quality signal, which is the concern TIDE raises against blanket debiasing. We therefore check whether the projected-out direction correlates with average rating on datasets that have ratings, and we report how projection changes cluster coherence in metadata space. If $w$ mostly encodes quality, the projection should hurt metadata coherence, and the experiments will show it.
The audit stage uses the corrected labels to explain debiasing. For each labeled cluster we compute the cluster's share of top-20 recommendations and divide it by the cluster's share of held-out interactions on the uniform test splits used by MACR. We report this exposure-to-interest ratio before and after applying MACR, PAAC, Adap-τ and ICPE. This produces a statement such as "PAAC raises exposure of 'neighborhood bakeries' but leaves 'late-night bars' starved," which no current debiasing method provides. It also gives ICPE clusters, which are currently numeric index sets, a readable description, because the same labeler can be run on them.
The proposal differs from FairParTree, which builds fair rule-based trees, in what it protects. FairParTree enforces balance over a demographic attribute in tabular data. We treat interaction volume as the quantity a description must not be dominated by, and we target natural-language labels rather than rule trees.
Baselines
The naive reference is k-means on raw collaborative-filtering embeddings with LLM labels written from the ten items nearest each center. This is the default pipeline and our diagnosis target, and every other method must reduce $\Delta_c$ relative to it.
The strongest existing labeling method is a SynCID-style pipeline (SynCID). It generates LLM descriptors per item, filters members inconsistent with the cluster descriptor, and labels the cluster from its center. SynCID reports 72.89% accuracy on BANKING against 67.43% for IDAS, but that result is for intent discovery on utterances. We found no reported result on Lacuna for cluster labeling of recommender item embeddings, so we run SynCID ourselves. Its filtering step may actively discard tail items as outliers, which makes it both a strong competitor and a direct test of our hypothesis. Beating it means lower $\Delta_c$ and higher tail recall at equal or better head recall, together with a better score in a blinded human preference study on 50 clusters per dataset.
Two exemplar baselines get the same LLM, the same number of exemplars (ten), the same prompt template and the same number of revision rounds as our method. One draws exemplars uniformly at random within each cluster. The other stratifies them by popularity quartile.
Three ablations isolate the ingredients. The first applies geometry correction alone, with center exemplars. The second applies exemplar selection alone, on raw embeddings. The third runs the full pipeline without the verification loop. The key test is whether geometry correction contributes beyond exemplar sampling.
Why not just…?
Why not just sample exemplars uniformly from each cluster? This is the most serious threat. If clusters are semantically sound and only exemplar choice is biased, uniform sampling should close most of the gap, since tail items already make up most of each cluster's members. Our hypothesis says it will not close the gap, because the clusters themselves partly encode popularity, as PAAC found in embedding space. The uniform-sampling baseline and the geometry-only ablation settle this directly. If uniform sampling matches the full method on $\Delta_c$ and tail recall, we will drop the geometry correction and report the simpler result.
Why not just cluster items on their text metadata and ignore the recommender's embeddings? Metadata clusters avoid popularity skew, but they describe the catalog, not what the recommender learned. An audit of exposure has to name the regions the model actually groups together. We include metadata clustering as a reference for label quality, and we measure how much its clusters disagree with the recommender's clusters. If the two agree closely after our correction, metadata clustering becomes a cheaper substitute for the audit, and we will say so.
Why not just use TF-IDF keywords from member items? Keyword lists are hard to read (Singh et al.; Schneider et al.). In interaction data they are also weighted toward head items, because popular items contribute more reviews and therefore more tokens. We run both a review-weighted and an item-weighted version through the same retrieval test. If the item-weighted version is competitive on tail recall, it becomes the cheap default and the LLM's contribution is limited to readability.
Why not just apply a debiasing method and skip labeling? Debiasing methods report aggregate gains: 30.2% in NDCG@20 on Yelp2018 for PAAC, and an average 153% in HR@20 over matrix factorization on uniform test sets for MACR. They do not say which concepts gained. The audit is meant to complement these methods, not compete with them. Its value depends on the predictive check described below.
Proposed Experiments
The first decisive check is the diagnosis. On Yelp2018, Amazon-book, MovieLens and Last.fm with matrix factorization and LightGCN, does the naive pipeline show a consistently positive $\Delta_c$, and is it larger for LightGCN, as the hub-amplification finding of MACR predicts?
The second check uses synthetic ground truth. We pick a known metadata category, such as one Yelp business category or one MovieLens genre, and downsample its interactions so that its items become tail items. We then retrain the model and ask whether each labeler still names that concept in the cluster that contains it. Recovery rate across 20 injected categories is the main comparison against the uniform-sampling and SynCID baselines.
The third check tests whether the audit predicts anything. For each debiasing method, the clusters our audit flags as starved before debiasing should be where that method's gains on the uniform test split concentrate. We measure this with rank correlation between pre-debiasing exposure-to-interest ratios and per-cluster HR@20 improvement. A labeling scheme that is truly popularity-aware should give stronger correlation than one built on naive labels. If it does not, the audit adds description without diagnostic value, and we will report it that way.
References
- Adap-τ : Adaptively Modulating Embedding Magnitude for Recommendation
- Popularity-Aware Alignment and Contrast for Mitigating Popularity Bias
- Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System
- ICMT: Item Cluster-Wise Multi-Objective Training for Long-Tail Recommendation
- Synergizing Large Language Models and Pre-Trained Smaller Models for Conversational Intent Discovery
- Hierarchical Level-Wise News Article Clustering via Multilingual Matryoshka Embeddings
- Popularity Bias Is Not Always Evil: Disentangling Benign and Harmful Bias for Recommendation
- Balancing Fairness and Interpretability in Clustering with FairParTree
- FIDELITY: Fine-grained Interpretable Distillation for Effective Language Insights and Topic Yielding
- Efficient and Flexible Topic Modeling Using Pretrained Embeddings and Bag of Sentences