Popularity Bias as Miscalibration in Early Popularity Forecasts

Platforms routinely watch the first hour or day of engagement on a new post and forecast whether it will "break out," meaning it ends up among the most-viewed items in its category. They then use that forecast to decide what to promote. This proposal argues that the popularity bias of these forecasters usually does not appear as bad ranking. It appears as dishonest probabilities that depend on how popular the item's creator already is.

The claim is that a forecaster can be accurate on average and still, when it says "30% chance of breakout," mean something closer to 15% for a video from a large channel and 45% for one from a small channel. A promotion rule that thresholds on this number would then systematically favor established creators. We propose to measure this effect using calibration conditioned on both early speed and creator popularity. We will then test whether recalibrating within those groups removes it, and whether doing so changes which new items a cost-aware promotion policy would pick.

The motivation comes from two findings that have never been put together. First, early-stage forecasting is mostly a speedometer. Predictability of Popularity: Gaps between Prediction and Understanding shows that on Last.fm, Twitter, Flickr and Goodreads, the time between an item's 1st and 5th adopter carries nearly 97% of full-model predictive power. Accuracy falls below 65% once items are matched on that speed. The authors read early speed as a measurement of cumulative advantage (the rich-get-richer loop) rather than of content appeal, but they report only accuracy and AUROC and never ask whether the probabilities are honest. Second, calibration research repeatedly finds that errors hide inside subgroups and over time while ranking metrics look fine. The clinical temporal-shift review Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine reports calibration deteriorating in 11 of 15 studies while discrimination held. Algorithmic decision making and the cost of fairness shows that group-level criteria can conflict with pooled calibration. If early velocity partly encodes a creator's existing audience, pooled training should produce exactly this kind of hidden, popularity-aligned miscalibration.

Core Claim

We make one hypothesis and one method claim. The hypothesis is that early popularity forecasters trained on pooled data show systematic calibration error that tracks prior creator popularity. They are overconfident for items from head creators and underconfident for tail creators, and the gap is largest among items with the same early speed, which is precisely where the matched-speed analysis of Shulman et al. says the forecaster understands least.

The method claim concerns a specific requirement. We ask that the forecast $\hat p$ of breakout $Y$ satisfy

$$P(Y=1 \mid \hat p = p,\ S=s,\ G=g) = p \quad \text{for all speed strata } s \text{ and creator-popularity groups } g.$$

Here $S$ bins the observed early speed (for example the Day-1 engagement level on SMTPD, or time-to-5th-adopter elsewhere), and $G$ bins the creator's prior audience size. Recalibrating to this requirement is the right intervention, rather than debiasing the ranking score. It keeps the prediction target honest, namely the observed outcome. It also stops creator popularity from silently inflating the number a promotion rule acts on.

The ingredients are not new: stratified recalibration, and threshold policies that are optimal under calibration. What is new is using speed-matched calibration as the lens for popularity bias in forecasting, and showing, or refuting, that it changes promotion decisions.

This is deliberately different from the closest bridge papers. Uncertainty Calibration for Counterfactual Propensity Estimation in Recommendation calibrates static exposure propensities so that inverse-propensity debiasing works. It has no temporal cascade and no early observation window. Long-Tail Session-based Recommendation from Calibration uses "calibration" to mean matching the share of tail items in a list via KL divergence, and never measures probability calibration. We work on the forecasting side, with frequency calibration, conditioned on early speed.

Research Program

The first component is a diagnostic protocol that turns the matched-speed analysis of Shulman et al. from an accuracy test into a calibration test. For each dataset we define breakout as exceeding the 90th percentile of final popularity within the item's category among training-period items. We fit forecasters on early-window features, then report reliability within cells of (speed stratum × creator-popularity group). Breakouts are rare and cells are small, so we use adaptive equal-mass bins and bootstrap confidence intervals, and we report the reliability and resolution terms of the Brier score. Two quantities matter most. The resolution term inside a speed stratum is the forecaster's knowledge beyond the speedometer. The reliability term inside each creator group is its popularity-aligned dishonesty.

The second component is recalibration. We compare three options on a held-out calibration split:

  • one pooled isotonic or Platt map;
  • separate isotonic maps per cell;
  • a boosting-style group recalibration that iteratively corrects the forecast on any cell, including intersecting cells, whose mean residual exceeds a tolerance.

The third option is meant to share strength where per-cell isotonic runs out of positive examples, such as the "tail creator, fast start" cell. We also run the same recalibrators on a rolling window over posting time, since calibration is the part known to decay under shift.

The third component is decision evaluation. Forecasting Model Success at Inference Time: Calibrated Probabilistic Forecasts for Cost-Optimal LLM Cascades proves that a fixed threshold on calibrated failure probabilities is cost-optimal. Its isotonic map cut expected calibration error from 0.12 to 0.03 for LLM routing. We borrow that argument for promotion: promote an item when $\hat p$ exceeds a cost ratio. We then measure realized decision cost on held-out outcomes, overall and per creator group, along with the share of promotion slots going to tail creators.

We are explicit that this is offline. Promotion's own causal effect on the outcome is not identified, so we measure decision quality against the outcomes that actually occurred, not a counterfactual world.

Baselines

The naive reference point is a speed-only forecaster: logistic regression on the single early-speed feature, calibrated with pooled isotonic regression. Shulman et al. show this feature captures nearly all predictive power. Any richer model must therefore beat it on within-stratum resolution, not just overall AUROC. By construction it has zero within-stratum resolution, which makes it the right yardstick for "understanding beyond the speedometer."

The strongest existing forecaster is the SMTPD temporal model from SMTPD: A New Benchmark for Temporal Prediction of Social Media Popularity. It combines ResNet-101 cover-frame features, multilingual BERT text features, and an LSTM over the popularity sequence, trained with a Composite Gradient Loss, and it uses Day-1 early popularity. It reports Day-30 MAE of 0.732 and Spearman rank correlation of 0.959. We will retrain it with its published configuration on our temporal split and attach a breakout head. For its breakout probabilities or calibration error, no reported result was found on Lacuna, so we will establish that number ourselves. It is the strongest competitor because it already uses both early signals and content, so any popularity-aligned miscalibration we find in it cannot be blamed on a weak model.

The third baseline is the same SMTPD model given creator prior-popularity features and a generous tuning budget. All variants get the same data, the same temporal split, the same hyperparameter search size, and the same calibration split. Beating it means two things together. First, per-group calibration error, averaged across creator groups with confidence intervals, drops significantly relative to both pooled isotonic and the feature-augmented model. Second, overall AUROC and Spearman correlation stay within noise and realized promotion cost does not rise.

The key ablation removes the speed strata from the grouping, calibrating by creator group only. This isolates whether conditioning on early speed, the proposal's central ingredient, matters.

Why not just…?

Why not just add creator popularity as a feature and retrain? A flexible model that sees the group variable can in principle become calibrated within groups, so this is the most serious threat. Pooled training still minimizes average loss, and heavy-tailed data means head-creator items dominate the gradient, so we expect residual per-group error. That is an expectation, not evidence. The feature-augmented baseline above tests it directly. If it matches group recalibration, we will report that popularity-aligned miscalibration is cheap to fix. The diagnostic protocol and decision analysis remain the contribution in that case.

Why not just apply one pooled isotonic map, as in the LLM-cascade work? Pooled recalibration guarantees only average calibration. The group-conflict results in Algorithmic decision making and the cost of fairness and the temporal-shift review both indicate that errors can survive inside subgroups. Our per-cell reliability measurements show directly whether pooled isotonic leaves a creator-group gap.

Why not just debias the score with counterfactual subtraction, as in MACR? Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System subtracts the estimated direct item-popularity effect from ranking scores and reports large HR@20 gains (153% for matrix factorization, 241% for LightGCN). However, it changes what is predicted and treats popularity as a static count. The subtracted score is no longer a probability of any observable event, so it cannot feed a cost threshold. We include a MACR-style variant, which subtracts a creator-popularity main effect from the forecast logit, and evaluate it on the same calibration and decision metrics.

Why not just use per-cell isotonic regression and stop? If it matches the boosting-style recalibrator, we will adopt it. We expect it to fail only in sparse cells, such as fast-starting tail items with very few breakouts. We will report per-cell sample sizes so readers can see where that happens.

Proposed Experiments

SMTPD is the primary testbed. It has 282,400 YouTube videos from 152,700 users with daily popularity for 30 days, and its user identifiers let us compute each creator's prior upload popularity from earlier posts only. We split by posting time and use later months for testing and rolling recalibration. We will attempt the same diagnostic on the Last.fm, Twitter, Flickr and Goodreads cascades used by Shulman et al., where time-to-5th-adopter is available directly. We do not know whether these datasets can be obtained in the same form.

As an exposure check, we will look at KuaiRand, which has a missing-at-random test set per Uncertainty Calibration for Counterfactual Propensity Estimation in Recommendation. If its randomly exposed portion supports item-level early windows, it lets us ask whether the creator-group gap shrinks when exposure is randomized. Shrinkage would be evidence that the gap comes from algorithmic amplification rather than genuine differences in appeal.

The decisive checks are three.

  • Hypothesis check: Does the SMTPD forecaster, after pooled isotonic recalibration, show a creator-popularity calibration gap whose confidence interval excludes zero, and is that gap larger within speed strata than overall? If not, the hypothesis is refuted and we report so.
  • Method check: Does group recalibration close the gap without losing Spearman correlation or AUROC, and does it beat the feature-augmented model?
  • Decision check: Under a threshold promotion policy, does group recalibration reduce realized cost for tail creators without raising total cost? And how does calibration error drift month over month with and without rolling recalibration?