When a creator's posts stop reaching people, there are two very different explanations. The platform may have quietly demoted them, lowering their reach in feeds and recommendations without telling them (often called shadow banning). Or the recommender may simply be doing what recommenders do: favoring items that already have many interactions, so that niche items in the "long tail" are rarely shown. This second effect is called popularity bias. Today an outside auditor cannot tell these two explanations apart. Regulators are starting to ask for audits of hidden demotion, yet Position: Machine Learning-powered Assessments of the EU Digital Services Act Aid Quantify Policy Impacts on Online Harms points out that transparency rules cover explicit removals but give no operational way to examine demotion inside recommendation back-ends. Without a principled way to separate the two causes, any demotion audit will either accuse platforms of censorship whenever tail items are under-exposed, or excuse real demotion as "just popularity."
We propose an audit test whose null hypothesis is "this item's exposure is what popularity bias plus ordinary training randomness would produce." The auditor trains a set of shadow recommenders on the same interaction data, varying random seeds and model families. This set yields, for every item, a distribution of exposures it could plausibly receive with no demotion. An item is flagged only when its observed exposure falls below that distribution. The test has two ingredients that existing tools lack. The first is that the reference distribution reproduces architecture-specific popularity amplification, which a simple popularity regression does not capture. The second is that it includes seed-to-seed arbitrariness, which we expect to be largest exactly in the long tail, where accusations of demotion matter most.
Core Claim
A demotion audit is only as credible as its model of what would have happened without demotion. We claim that this null model must reproduce two things. The first is the popularity amplification of the recommender family in question: Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System shows that graph-based recommenders such as LightGCN amplify popularity far more than matrix factorization, because information spreads through high-degree hub items. The second is the predictive multiplicity of equally accurate models. Algorithmic Arbitrariness in Content Moderation found that retraining a toxicity classifier with a different seed flips 34.2% of decisions on ToxiGen and 23.9% on Jigsaw. That paper studied text classifiers only. Nobody has measured the analogous lottery for long-tail exposure in recommenders.
Our concrete predictions follow from these two requirements. A test that conditions only on an item's own popularity will over-flag tail items that sit near hubs in graph recommenders. A test built from a single shadow model will mistake seed noise for demotion. A seed-and-architecture envelope will hold its false-flag rate near the nominal level and still detect injected demotions of moderate strength. If the simple popularity regression performs as well, that is itself the finding, and we will report it as such.
Relation to Prior Work
Two lines of work motivate the gap and one supplies tools. On the auditing side, Setting the Record Straighter on Shadow Banning crawled 2.5 million Twitter accounts and rejected the platform's "random bug" explanation with probabilities as low as $10^{-315}$. It also showed that bans spread across neighbors: a banned neighbor raised a user's ban probability nearly 4-fold. That null hypothesis was uniform randomness, not popularity bias, and the paper never modeled the recommender itself. We keep its hypothesis-testing stance and replace its null with a recommender-aware one. The topological face of recommendation: models and application to bias detection detects items the platform injected into recommendations by finding long-range shortcut edges via edge-betweenness. It recovered nearly 50% of the biased YouTube links by inspecting the top 12.5% of edges. That method targets added promotion edges. Demotion removes edges, which is a different signal, so we include this method as a baseline rather than assume it transfers.
On the debiasing side, three methods correct popularity bias in different ways. MACR removes the direct effect of item popularity on the score using a causal graph. Adap-τ : Adaptively Modulating Embedding Magnitude for Recommendation controls how the length of an embedding vector grows with interaction count. Popularity-Aware Alignment and Contrast for Mitigating Popularity Bias, called PAAC, anchors tail items to head items they are co-interacted with. All three treat under-exposure of the tail as an unintended artifact to be erased. We use them in two roles. They serve as candidate "platform" models whose exposure the auditor must explain. They also appear inside the shadow ensemble, because a real platform may already debias. We do not use them as the reference for what an item "deserves." A platform that has popularity bias is not violating anything by having it, and an audit against a debiased reference would flag the entire tail.
A third finding constrains the design. Breaking Feedback Loops in Recommender Systems with Causal Inference stresses that exposure changes future interactions. A demoted item therefore accumulates fewer interactions over time, and its demotion gradually becomes indistinguishable from low popularity. We treat this as the hardest regime and test it explicitly.
Research Program
The auditor observes two things: the interaction log, and the platform's top-$K$ recommendation slates for a sample of users. This corresponds to data access for vetted researchers combined with sockpuppet-style querying. From the slates we compute the observed exposure $E_i$ of item $i$, meaning the number of sampled users whose top-$K$ list contains $i$. The auditor then trains $M$ shadow recommenders on the same log. These span seeds, model families (matrix factorization and LightGCN), and debiasing variants (none, Adap-τ, MACR, PAAC). Each shadow model is kept only if its validation accuracy lies within a tolerance of the best, which defines an empirical Rashomon set. Each shadow model $m$ produces an exposure $E_i^{(m)}$. The per-item audit statistic is
$$p_i = \frac{1 + \#\{m : E_i^{(m)} \le E_i\}}{M + 1},$$
which is the empirical probability that a demotion-free model gives item $i$ exposure at least as low as observed. Because the platform's own model differs from every shadow model, we first calibrate exposure scale with a monotone mapping fitted on items presumed clean. We then control the false discovery rate across items with Benjamini-Hochberg.
The program has three parts. The first measures the exposure lottery: how much $E_i$ varies across equally accurate recommenders, by popularity decile and model family. No reported result for seed-induced exposure multiplicity in recommenders was found on Lacuna, so this measurement stands on its own. The second builds and evaluates the test on platforms with known demotions planted in them. The third stresses the test under two conditions: the shadow ensemble omits the platform's true architecture, and demotion is applied over several feedback rounds.
Baselines
The naive reference flags items with the lowest ratio of exposure to interaction count. This is what an auditor with no recommender model would do, and it shows how badly the tail is over-flagged when popularity bias is ignored.
The strongest simple competitor is a popularity-residual regression. We fit observed exposure against log interaction count and a few neighborhood statistics, such as mean neighbor degree and co-interaction count, using a flexible regressor. Items are then flagged by their standardized negative residual. This is the test a careful skeptic would build, and it already encodes some hub effects through its features. It receives the same log, the same slates, and the same feature-tuning budget that our method spends on its ensemble.
The topological audit of The topological face of recommendation is run on the item-to-item recommendation graph induced by the platform's slates. Items are scored by how much betweenness centrality is missing relative to the shadow models' graphs. Its reported result is the nearly-50%-at-12.5% figure for injected YouTube shortcuts. No reported result exists for demotion detection on Lacuna.
The debias-then-compare baseline uses MACR, a strong debiaser. MACR's reported gains of 153% (matrix factorization) and 241% (LightGCN) in HR@20 are averages over five datasets. The baseline flags items whose MACR-predicted rank exceeds their platform rank by the largest margin.
Two ablations isolate our ingredients. The first is a single-seed shadow model of the correct architecture, which removes multiplicity. The second is a seed-only ensemble restricted to the wrong architecture, which removes architecture coverage. As an upper bound, we use an oracle ensemble containing the platform's exact architecture and training recipe.
Our method beats the field if, at matched false-flag rates on undemoted tail items, its area under the ROC curve for detecting demoted items exceeds the popularity-residual regression in the bottom three popularity deciles on LightGCN platforms, and it does not lose on matrix factorization platforms. No prior paper reports a number on this exact task, so every comparison is run fresh under identical inputs.
Why not just…?
Why not just regress exposure on popularity? This is the real threat. On matrix factorization platforms, where exposure is driven mostly by an item's own embedding length, as Adap-τ documents, a regression on interaction count may match our test. We expect it to fail on graph-based platforms, where MACR shows that hub neighborhoods shape tail exposure, and in deciles where seed noise is large. A regression has a single residual variance and cannot widen its tolerance for items whose exposure is a lottery. The decile-stratified false-flag comparison settles the question. If the regression matches us everywhere, we will report that cheap audits suffice.
Why not use a debiased recommender as the reference? MACR, Adap-τ, and PAAC estimate what an item would get without popularity bias. A platform is not obliged to provide that. Comparing against such a reference measures the Matthew effect, the tendency of already-popular items to gain more, and not demotion, so it should flag most of the tail. The debias-then-compare baseline tests this directly through its false-flag rate on undemoted tail items.
Why not use the topological shortcut audit? That audit was designed and validated for edges a platform adds. Demotion deletes edges, and deleting edges from low-degree tail items barely changes betweenness centrality. We run it as a baseline. If it detects contagion-style clustered demotion well, which is plausible because clusters of removed edges do reshape paths, we will add its score as a feature in our test.
Why not train one shadow model of the right architecture? The arbitrariness findings from text moderation suggest that one seed is an unreliable null. The single-seed ablation measures how many false accusations this produces.
Proposed Experiments
We use Yelp2018, Gowalla, Amazon-book, and MovieLens-1M, the benchmarks shared by MACR, Adap-τ, and PAAC. Gowalla is included specifically because it is the sparse case: PAAC reduced the popular-versus-unpopular accuracy gap by only 34.18% on Gowalla versus 87.5% on Yelp2018. On each dataset we train platform models: matrix factorization, LightGCN, and LightGCN with PAAC, where PAAC is the strongest debiased platform, with a reported +30.2% NDCG@20 on Yelp2018.
We then plant demotions on 1–2% of items. The demotion is applied either as a multiplicative score penalty or as a rank cap, at five strengths. Demoted items are chosen in three ways: uniformly at random, concentrated in the tail, or clustered in co-interaction neighborhoods. The clustered choice mimics the contagion pattern from the Twitter audit.
The primary metrics are three. We report detection AUROC and power at a 5% false discovery rate, stratified by popularity decile. We report the false-flag rate on clean tail items. And we report the minimum demotion strength detectable at 80% power per decile, which defines an "auditability floor."
Three decisive checks follow. First, if LightGCN's tail exposure varies across seeds by roughly as much as a moderate demotion, then single-model audits are unsound in the tail. That would be a publishable negative result regardless of how the rest turns out. Second, the misspecification test removes the platform's architecture from the ensemble. If false flags rise only modestly, the method is practical for auditors who do not know the platform internals. Third, a five-round closed-loop simulation regenerates interactions from exposure each round. We track how quickly a demotion's detection power decays as the demotion is absorbed into apparent unpopularity. This tells regulators how soon after a suspected demotion an audit must run.
References
- Position: Machine Learning-powered Assessments of the EU Digital Services Act Aid Quantify Policy Impacts on Online Harms
- Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System
- Algorithmic Arbitrariness in Content Moderation
- Setting the Record Straighter on Shadow Banning
- The topological face of recommendation: models and application to bias detection
- Adap-τ : Adaptively Modulating Embedding Magnitude for Recommendation
- Popularity-Aware Alignment and Contrast for Mitigating Popularity Bias
- Breaking Feedback Loops in Recommender Systems with Causal Inference