When the Quality Judge Is Swayed by Popularity: Testing Anti-Monopoly Guarantees on Real Review Data

Online markets tend to concentrate. A few early winners take most of the attention because popularity signals feed back into what people see next. Two theoretical results say this can be tamed. One shows that if the platform shows a dampened popularity signal, growing slower than linearly with market share, the market settles into a stable ranking that tracks true quality. The other shows that splitting the market into independent sub-markets breaks winner-take-all dynamics. Both results assume the platform already knows each item's true quality. In practice, quality is estimated by people or models: reviewers, editors, or increasingly LLM judges. Those evaluators are themselves swayed by visibility. The clearest example is that ICLR papers posted on arXiv before review were accepted at 48.16% versus 28.94% for anonymous ones.

This proposal asks what happens to the anti-monopoly guarantees when the quality estimate is contaminated by the same popularity it is supposed to counterbalance. Our claim is that contamination adds directly to the effective strength of the popularity feedback. As a result, a dampening setting that is provably safe under known quality can still lock in a monopoly once quality is estimated. We will measure that contamination on real peer-review data and on LLM judges. We will then test whether the structure of an editorial board can bring the guarantee back: blind lower-tier reviewers, expert-weighted aggregation by area chairs, and sub-markets defined by area-chair pools.

Core Claim

The starting point is Popularity signals in trial-offer markets with social influence and position bias. This paper models a trial-offer market, in which people first choose what to try and then buy with probability equal to the item's quality. It proves a sharp threshold. When the popularity signal $f(x)=x^r$ uses an exponent $r<1$, the market converges to a unique equilibrium that reflects quality. When $r>1$, it converges to an unpredictable monopoly. We take its model unchanged except for one point: the platform no longer sees true quality $q_i$. It sees an estimate $\hat q_i = q_i\,\phi_i^{\beta}\,\epsilon_i$, where $\phi_i$ is item $i$'s current market share, $\beta \ge 0$ is the evaluator's sensitivity to visibility, and $\epsilon_i$ is independent noise.

Suppose this estimate enters the probability of trying item $i$ multiplicatively, as it does when the platform weights items by score. Then the attraction of item $i$ scales as

$$ \Pr(\text{try } i) \;\propto\; a_i\, q_i\, \phi_i^{\,r+\beta}\,\epsilon_i , $$

where $a_i$ is item appeal. The safe region therefore shrinks from $r<1$ to

$$ r + \beta < 1 . $$

We will prove this in the multiplicative case by linearizing the paper's mean-field dynamics around equilibrium. For the cases where the estimate drives position ranking rather than weighting, we will test it in simulation. The practical consequence is concrete. A platform that picks $r=0.8$ on the strength of the published theorem is unsafe whenever its evaluators have $\beta>0.2$. Whether real evaluators reach that level is an empirical question that nobody has measured.

The editorial-board direction enters as the mechanism that controls $\beta$, not as decoration. Journal and conference hierarchies already separate atomic judgments from aggregation. Examples include the boards described in IEEE Transactions on Information Theory publication information and the VLDB front matter. In these structures, reviewers score, associate editors or area chairs reconcile scores within a topical pool, and executives set policy. Each tier is a natural place to intervene. Reviewers can be blinded to visibility, area chairs can down-weight visibility-sensitive reviewers, and area-chair pools can act as the independent sub-markets of segmentation.

Relation to Prior Work

Two papers supply the mechanisms we stress-test, and both motivate the gap. Maldonado et al. prove the $r$ threshold but validate it only on a synthetic 50-song MusicLab market with fixed, known quality. Taming the Matthew Effect in Online Markets with Social Influence introduces the Randomized Segmentation Protocol, which splits the market into $m$ independent worlds with perturbed quality signals. In stylized simulations it lowers inequality from 1.0 to about 0.45 at under 5% purchase loss. Neither paper models an estimated, visibility-contaminated quality signal. Neither uses real evaluation data. Neither considers who in an organization implements the mechanism.

Two empirical papers supply the data and constrain the design. What have we learned from OpenReview? documents three facts across 11,915 ICLR submissions: the arXiv acceptance gap, inflated scores from low-confidence reviewers, and rebuttals raising scores only 10.76% of the time. It is purely observational, however. Its arXiv gap mixes visibility bias with genuine quality differences, so it cannot be read directly as $\beta$. ReviewArena benchmarks LLM reviewers on 51,529 papers and finds two problems. Scores are compressed between 4 and 7, and the average accept/reject gap is only about 0.5 points. Compression matters for us because it shrinks the true-quality component of $\hat q$ relative to any visibility component. ReviewArena never tests whether LLM judges respond to visibility cues.

Trust Models and Con-Man Agents shows a related failure in reputation systems: trust models that are sound on paper break once agents adapt to how evaluation works. We read it as a warning that guarantees resting on an idealized evaluator need adversarial testing. Embodied LLM Agents Learn to Cooperate in Organized Teams finds that a designated leader improves LLM team efficiency, for example 9.76% faster task completion for three GPT-3.5 agents. That result is from household chores, so we treat it only as weak motivation that hierarchy helps LLM agents. It is not evidence about evaluation.

Research Program

The work has three parts.

Part one: theory and simulation. We extend the Maldonado trial-offer simulator so that the platform's quality signal is produced by an evaluator with sensitivity $\beta$ and noise level matched to real reviewers. We then map the stability region over $(r,\beta)$ under both quality-weighting and position-ranking displays. We will prove the $r+\beta<1$ shift in the multiplicative case and report where simulation departs from it.

Part two: measuring $\beta$ for human and LLM evaluators. For humans we use the ICLR OpenReview corpus. We cannot identify a single absolute $\beta$, because arXiv-posted papers may simply be better. We can, however, identify the difference in visibility sensitivity between reviewer types on the same paper. The method is a within-paper difference-in-differences: compare expert and novice reviewers on papers that were and were not on arXiv, with a paper fixed effect absorbing true quality. Citation counts at decision time serve as a second, continuous visibility measure. The absolute level is bounded by a sensitivity analysis rather than claimed as point-identified.

For LLM judges we can intervene directly. On the 1,002-paper ReviewArena-Eval subset, we take each paper and vary only an injected visibility cue: a stated citation count, a stated download count, or a note that the paper is a widely discussed preprint. We then regress scores on the log of the cue. This gives a causal $\beta$ for each model, including the best-calibrated ones.

Part three: governance variants. We plug the measured $\beta$ values and fitted reviewer-noise models into the simulator. The noise models come from a hierarchical fit on ReviewArena, with a paper effect plus reviewer-confidence-dependent bias and noise. We then compare editorial structures that each act on a different term. The first is a blind reviewer tier, which lowers $\beta$ at the source at some leakage rate estimated from the arXiv data. The second is area-chair aggregation that weights reviewers by their estimated visibility sensitivity, which lowers the effective $\beta$ of the aggregate. The third is segmentation along area-chair topical pools instead of random worlds, which keeps each sub-market's popularity independent while keeping items that genuinely compete together. Our hypothesis is that topical segmentation matches Randomized Segmentation's inequality reduction with lower purchase loss, because random worlds waste attention by mixing unrelated items.

We state the empirical limit plainly. The attention market of readers choosing among accepted papers is simulated. What the real data contributes is the evaluator: its noise, compression, and visibility sensitivity. This is precisely the component the existing theorems idealize away.

Baselines

Naive reference. Linear popularity ranking ($r=1$) with no correction, run in two versions: one using true quality and one using the estimated $\hat q$. This is the uncontrolled platform and anchors the inequality scale.

Strongest dampening method. Quality ranking with sublinear signaling from Maldonado et al., run with true quality as the oracle and with contaminated $\hat q$ at the measured $\beta$. The file reports only that sublinear markets reached a stable equilibrium within about $10^4$ iterations and were significantly more predictable than linear or superlinear ones. Its exact inequality and efficiency numbers are not available here, so we rerun it ourselves in the same simulator.

Strongest segmentation method. Randomized Segmentation from Berbeglia and Van Hentenryck, whose reported result is inequality of about 0.45 against 1.0 for quality ranking, at a purchase loss ratio under 5%. We run it with the same item pool, number of consumers, random seeds, and an equal tuning budget over $m$ and noise level $\sigma$ as every governance variant receives over its own parameters.

Ablation of the key ingredient. Area-chair aggregation with the visibility-sensitivity weighting turned off, so that it averages reviewers uniformly. This isolates whether the gain comes from correcting $\beta$ or merely from aggregating more scores.

Statistical debiasing. A single regression that subtracts estimated $\beta \log \phi$ from every score, with no hierarchy at all.

A governance variant beats the strongest baseline if it meets two conditions on the measured-$\beta$ setting, across at least 50 seeds. Its rank correlation between final market share and latent quality must be higher. Its inequality must be no higher than Randomized Segmentation's at equal or lower purchase loss. For LLM judges, the reference calibration figures are ReviewArena's best mean absolute error of 0.700 (Qwen3.6-Plus) and the largest accept/reject gap of 1.40 points (GLM-5.1). We will report each judge's $\beta$ alongside these, to test whether better calibration predicts lower visibility sensitivity.

Why not just…?

Why not just set $r$ very low, or turn off the popularity signal entirely? That removes the feedback, but it also discards information. Social signals help people find good items faster, which is why Maldonado et al. compare efficiency in total purchases and not only stability. Our simulation reports the efficiency-versus-$r$ curve. If the curve is flat near $r=0$, this alternative wins and the contribution reduces to the measurement of $\beta$. We expect a real cost, which would make knowing $\beta$ necessary for choosing the largest safe $r$.

Why not just blind the evaluators? Blinding is one of our arms, but the ICLR arXiv gap is itself evidence that double-blind review leaks. LLM judges may also infer prominence from writing style or topic. The prompt-intervention experiment measures $\beta$ under explicit cues. A paraphrase variant that removes the cue but keeps the text tests whether the LLM's sensitivity persists without it.

Why not just use Randomized Segmentation, which never needed quality to be known? This is a real threat. Segmentation keeps each world's popularity independent, but within each world the contaminated estimate still compounds. Our $(r,\beta)$ map run under segmentation will show whether segmentation alone stays stable at the measured $\beta$. If it does, we will say so, and the topical-pool variant must then win on purchase loss to justify itself.

Why not just regress out visibility in one statistical step instead of building a hierarchy? If the single-regression baseline matches area-chair aggregation, the honest conclusion is that what matters is measuring and subtracting $\beta$, not the hierarchy. The hierarchy can only add value through reviewer-specific sensitivity, which a pooled regression averages away. The within-paper expert-versus-novice difference is precisely the quantity that tells us whether that heterogeneity is large enough to matter.

Decisive Checks

The idea stands or falls on four results. First, the simulated stability threshold should move to approximately $r=1-\beta$ as $\beta$ increases. If it does not move, the core claim is wrong. Second, the measured visibility sensitivities, both the expert-versus-novice difference on ICLR and the causal $\beta$ of LLM judges on ReviewArena-Eval, must be large enough to make commonly chosen sublinear exponents unsafe. If they are near zero, the problem is moot in practice, and we will report that as a useful negative validation of the existing theorems. Third, at the measured $\beta$, some governance variant must recover quality-tracking stability where uncorrected dampening fails. Fourth, the ablations must attribute the recovery to the visibility-sensitivity correction rather than to aggregation alone.