Keeping Popularity Out of the Picture: Per-Modality Popularity Erasure for Precomputed Multimodal Recommendation Indexes

Many modern recommenders describe each item with a product photo and a text description. Each is passed through its own encoder, called a "tower," and the results are merged into one vector that is computed once, offline, and stored in a search index. A recent audit found that these two inputs do not behave alike. Text features raise accuracy with little side effect. Visual features act as a popularity amplifier: they push recommendations toward already-popular items and shrink the number of distinct items that ever get recommended (On Popularity Bias of Multimodal-aware Recommender Systems). That audit stops at diagnosis. Existing popularity-debiasing methods also treat the item vector as one undivided whole, and so do existing fixes for "hubness," where a few gallery items end up as nearest neighbors to far too many queries. None of them asks which tower is carrying the popularity signal.

We propose to find where popularity lives inside a late-fusion item vector, remove it from the tower that smuggles it in, and move it into a single explicit coordinate of the index. The serving system then controls that coordinate with one scalar weight. The index remains a plain set of static vectors, so fast approximate nearest-neighbor search still works. The popularity preference becomes an adjustable setting instead of a side effect of how product photos were encoded.

Core Claim

In a late-fusion recommender, the item vector is a sum of parts: $v_i = f_{\text{img}}(x^{\text{img}}_i) + f_{\text{txt}}(x^{\text{txt}}_i) + e_i$. Here $f_{\text{img}}$ and $f_{\text{txt}}$ are the trained projection towers over frozen image and text features, and $e_i$ is an optional learned per-item ID embedding. Because the vector is a sum, each tower's share of the ranking score $u^\top v_i$ (with $u$ the user vector) can be measured separately.

Our hypothesis is specific. The visual tower learns a direction in the shared space along which log-popularity can be linearly read off. Head items often share a professional photographic style, so image features give the model a cheap proxy for "this is a bestseller." User vectors then align with that direction for everyone.

The proposed fix has two parts, both applied during training. First, a linear-erasure penalty strips popularity information from each modality tower's output. Second, an explicit popularity channel absorbs the removed signal. The indexed item becomes $[\,v_i;\ b_i\,]$, where $b_i$ is a learned scalar fed by item log-popularity. The online query becomes $[\,u;\ \lambda\,]$, where $\lambda$ sets how much popularity counts at serving time.

The erasure penalty for tower $m$ over a batch is the squared norm of the cross-covariance between the tower output $z_m$ and log-popularity $p$, plus a term on the correlation between the output's length and popularity:

$$\mathcal{L}_{\text{erase}}^{(m)} = \big\|\widehat{\mathrm{Cov}}(z_m, p)\big\|_2^2 + \mu\,\widehat{\mathrm{Corr}}\big(\|z_m\|_2, p\big)^2 .$$

The first term removes any linear direction that predicts popularity. The second targets the length-inflation mechanism that Adap-τ and Quantifying and Mitigating Popularity Bias in Conversational Recommender Systems identified for inner-product scoring: frequently seen items get longer vectors. The penalty is closed-form and cheap. Its per-tower weight is set from a diagnostic measurement rather than tuned blindly.

What would be new is the combination of two things. The debiasing acts on individual modality towers, guided by a measurement of which tower carries popularity. And the output is an index that stays compatible with inner-product search while exposing popularity as a single controllable coordinate.

Relation to Prior Work

The modalities audit (On Popularity Bias of Multimodal-aware Recommender Systems) motivates this work. It tested VBPR, MMGCN, GRCN and LATTICE on three Amazon categories, found that the visual modality amplifies popularity, and found that the most accurate model, LATTICE, amplifies it most. It proposes no remedy and never looks inside the fused vector. We start from its setting, with frozen pre-extracted features, and go one level deeper, to tower-level attribution.

Three training-time debiasing methods supply techniques and serve as competitors. MACR models popularity as a direct causal effect of the item on the score and subtracts that effect at inference. PAAC uses popular items as anchors for unpopular ones and reweights contrastive negatives so the two groups stop occupying separate regions of the space. CLRec shows that the InfoNCE contrastive loss with in-distribution negatives acts as implicit inverse-propensity weighting. All three were built for ID-based collaborative filtering and treat the item representation as one block. Our explicit coordinate $b_i$ borrows MACR's idea of an isolated popularity path, but places it inside the index.

We also correct a claim in our own research notes. MACR's subtraction does not by itself rule out fast vector search: when the item-only term is a per-item scalar, it can be folded into an augmented inner product. The cost of serving is therefore not the real gap. The real gap is that no existing method controls where popularity enters the item vector.

The hubness papers constrain the design. Sinkhorn Normalization / DBSN and DBNORM rebalance similarity scores after training in dual-tower cross-modal retrieval. DBNORM itself reports that reducing hubness does not track retrieval quality linearly. Two Effects, One Trigger warns that forcing geometric changes after training, such as closing the gap between image and text embedding regions, damages performance. That paper also shows the gap sits in a few specific dimensions and plays a useful role. This is why we erase only popularity-predictive directions, during training, and not the image–text gap. It is also why the evaluation must confirm that the gap and accuracy survive.

Research Program

The first stage is a diagnostic, and it decides whether the rest of the project goes ahead. We train VBPR and LATTICE, with and without ID embeddings, on Amazon Office, Toys & Games, and Clothing, using the frozen features from the modalities audit. For each tower we then measure three things: how well a held-out linear probe predicts log-popularity from the tower output, how strongly output length correlates with popularity, and how much each tower's score share contributes to the average popularity of each user's top-10 list.

The premise holds only if the visual tower carries clearly more popularity information than the text tower, most of all on the sparse Clothing category. If the two towers carry similar amounts, the per-modality framing collapses. In that case the correct conclusion is that modality-agnostic debiasing suffices, and we report that outcome.

The second stage trains with the erasure penalty and the explicit popularity channel. The per-tower weights are proportional to each tower's measured probe accuracy, and we sweep $\lambda$ at serving time to trace an accuracy–coverage curve. The third stage checks that nothing important breaks. We confirm that linear popularity information has been removed; we then test whether a small nonlinear probe can still recover it, since linear erasure may leave nonlinear leakage. We also confirm that the image–text gap and fine-grained accuracy on head items stay intact.

Baselines

The naive reference is the unmodified LATTICE and VBPR backbones, with all modalities and plain inner-product scoring. It fixes the accuracy we must not lose. The strongest reported accuracy point in our materials is LATTICE at Recall@10 = 0.0664 on Office, from On Popularity Bias of Multimodal-aware Recommender Systems. We will reproduce it before comparing anything.

The strongest debiasing competitors are PAAC and MACR, applied to the fused item vector of the same backbones. They see the same frozen features, the same data splits, the same number of training epochs, and the same hyperparameter budget of 20 configurations each, selected on a popularity-debiased validation split. PAAC reports the largest reductions in the accuracy gap between popular and unpopular items, 87.5% on Yelp2018, so it is the method a skeptic would expect to match ours. We found no reported result on Lacuna for PAAC or MACR on multimodal Amazon benchmarks, so these numbers come from our own runs.

We also include the cosine normalization with adaptive temperature from Adap-τ, a text-only model, and the DBNORM and Sinkhorn post-processing applied to the final item index. The key ablations are three: the erasure penalty on all towers equally, on no tower (the explicit coordinate alone), and on the visual tower only. Together they isolate whether per-modality targeting matters.

To count as beating the baselines, our method must reach higher Average Percentage of Long Tail (APLT, the share of recommended items drawn from the long tail) and higher item coverage (iCov) at equal or better debiased Recall@10. Equivalently, its $\lambda$-swept accuracy–coverage curve must lie above the curves of PAAC and MACR applied to the fused vector.

Why not just…?

Why not just post-process the index with DBNORM or Sinkhorn normalization? These methods rescale scores against banks of example queries and gallery items. They leave the learned vectors unchanged, and DBNORM itself found that hubness reduction and quality do not move together. Sinkhorn normalization also adds iterative computation at query time, which conflicts with the reason for precomputing the index. Both remain real baselines on the final index, and the comparison at equal coverage will settle the question.

Why not just run PAAC or MACR on the fused vector? This is the real threat. If popularity enters through the fused vector no matter which tower supplies it, modality-agnostic debiasing should work equally well. The diagnostic stage and the "erase all towers equally" ablation are designed to show whether per-tower targeting gains anything. If they show it does not, we will say so.

Why not just drop the visual tower? The audit shows that text alone recovers most of the accuracy gain on these datasets. Dropping images may therefore be a strong baseline, and we include it. Our method must beat text-only on debiased recall, not merely on coverage, to justify keeping the visual tower.

Why not just cosine-normalize, which removes length inflation? Normalizing vector length removes only the length channel. Popularity can still be encoded in a direction, which is exactly what the probe measures. The Adap-τ baseline tests whether length accounts for most of the effect.

Proposed Experiments and Decisive Checks

Amazon has no uniformly sampled test set, so the evaluation follows the concern about biased test sets raised by MACR and PAAC. We build test splits reweighted by inverse popularity and stratified into popularity deciles, and we report Recall@10/20, APLT, iCov, and the recall gap between head and tail items.

Three results decide the question. First, the diagnostic must show that the visual tower carries more popularity than the text tower. Second, our curve must dominate fused-vector PAAC and MACR. Third, the visual-only ablation must recover most of the full method's gain.

As a small transfer check, we will apply the same erasure during fine-tuning of CLIP4Clip on MSR-VTT, using caption concept frequency in place of item popularity. We will compare the resulting hubness skewness and R@1 against DBNORM's reported improvement from 44.10% to 45.00% R@1. This check indicates whether erasing popularity during training generalizes beyond recommendation.