Subtracting the Crowd: Query-Marginal Rank Fusion for Community Answer Search

Search systems increasingly answer a query by running several different retrievers and merging their result lists. A common merging rule is reciprocal rank fusion (RRF), which gives each document the sum of $1/(k + \text{rank})$ over all retrievers, so documents ranked well by many retrievers rise to the top. The rule rests on an assumption: agreement between retrievers is evidence of relevance. That assumption fails when the retrievers share a bias. On a community Q&A site, a heavily viewed answer tends to be long, which helps a keyword retriever. It tends to sit near a large region of embedding space, which helps a dense retriever. It has many upvotes, which helps a metadata ranker. These retrievers then "agree" about the popular answer for reasons unrelated to the query, and fusion turns this shared bias into what looks like a strong consensus. The niche answer that exactly solves the question, found by only one specialized retriever, gets buried.

We propose a training-free fix, applied at the level of each retriever. For every document and every retriever, we estimate offline how much fusion credit that document collects when the queries are random and have nothing to do with it. That query-independent "pull" is the document's popularity effect inside that retriever. At query time we subtract it before summing. The correction is a precomputed lookup table, so it adds no query latency. It also targets the exact place where shared bias enters: each retriever's contribution to the fused score. We separately replace raw upvotes with an exposure-normalized quality signal, because a vote count mixes quality with how often the post was seen. The testbed is answer retrieval over Math StackExchange (the ARQMath lab). It has every ingredient: heterogeneous text, formula and dense retrievers, community metadata, a documented latency problem, and a heavy-tailed popularity distribution.

Core Claim

Our claim is that popularity bias enters rank fusion additively, once per retriever. It is therefore best removed per retriever, before fusion, rather than after fusion with one global penalty. For a query $q$ and document $d$, write $r_m(q,d)$ for the rank of $d$ in retriever $m$'s list. Let $\mathcal{Q}_0$ be a sample of queries drawn from the collection's own question titles, unrelated to $q$ by construction. Then the query-marginal fused score is

$$S(q,d) = \sum_{m \in M} \left[ \frac{1}{k + r_m(q,d)} - \beta\, \mathbb{E}_{q' \sim \mathcal{Q}_0}\!\left[\frac{1}{k + r_m(q',d)}\right] \right],$$

where $k$ is the usual smoothing constant and $\beta \in [0,1]$ controls how much correction is applied. A document absent from a list contributes zero. With $\beta = 0$ this is standard RRF. With $\beta = 1$ and no labels it is fully unsupervised.

The subtracted term plays the role of the "direct item effect" in causal debiasing. In Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System, MACR learns auxiliary branches that estimate an item's query-independent effect and subtracts it from a single recommender's score at test time. We keep that counterfactual subtraction but change where it is applied and how it is estimated. Instead of a learned branch inside one model, we use a Monte-Carlo estimate over counterfactual queries, taken separately for each retriever inside a fusion ensemble. This matters because the size of the overcount depends on how many retrievers share the bias. A single penalty on the fused list cannot know that number; a per-retriever correction accounts for it automatically.

The metadata channel needs different treatment, because an upvote ranker is entirely query-independent and the subtraction would erase it. We keep it as a channel that ranks only within each query's candidate pool. It ranks by votes per unit of exposure: votes divided by question views, where the collection provides them, or otherwise by post age and position in the thread.

Why This Gap Is Real

Several lines of work motivate the proposal.

First, rank fusion is known to be robust because it ignores incomparable score scales. Murat Akbacak et al. found rank-based fusion beats weighted score combinations in multimodal retrieval. Jing Lu et al. argue that needing no labels makes RRF robust under domain shift. Yuchen Hui et al. show that consensus across query reformulations filters spurious matches. All three treat consensus as a noise filter. None asks what happens when the noise is correlated across retrievers.

Second, the recommendation literature explains why that correlation should exist. Adap-τ shows that item embedding length grows with interaction frequency, so inner-product retrievers favor popular items whatever the query. Quantifying and Mitigating Popularity Bias in Conversational Recommender Systems finds the same length–frequency link in conversational systems.

Third, the metadata literature uses votes directly as quality signals. RedCaps filters Reddit data by upvote threshold. The ARQMath 2021 system of Zhong et al. feeds raw upvotes and accepted-answer flags into learning to rank. Neither separates quality from exposure.

The closest bridge paper is MMMORRF. It introduces weighted RRF with channel weights set by a metadata heuristic ("news-likeness"), reporting a 4.2% gain over plain RRF on MultiVENT 2.0. It reweights whole channels per query, but it has no document-level notion of shared bias. Our correction is document-specific and label-free, and it composes with weighted RRF.

Two other lines constrain the design. Towards Two-Stage Counterfactual Learning to Rank debiases a serial two-stage pipeline with click propensities, and it notes high estimator variance when candidate lists are small. We have no click logs on ARQMath, and variance problems argue against propensity weighting here. That is one reason we estimate the popularity effect from counterfactual queries rather than from inverse propensities. Long-Tail Session-based Recommendation from Calibration warns that blanket debiasing hurts users who genuinely want popular items. That warning is why $\beta$ is exposed and why we report head-query accuracy separately.

Research Program

We build four retrievers over the ARQMath Math StackExchange answer collection.

The first is BM25 over answer text in Anserini. The second is BM25 over operator-tree path tokens for formulas, following Zhong et al.; these tokens break a formula's tree structure into leaf-to-root path strings that an ordinary keyword index can handle. The third is a math-adapted ColBERT late-interaction retriever, following Applying Structural and Dense Semantic Matching for the ARQMath Lab 2022. "Late interaction" means each query token is matched to its best document token and the matches are summed. The fourth is the exposure-normalized vote channel, ranked within the candidate pool.

For the correction table, we sample 2,000 question titles as $\mathcal{Q}_0$ and exclude any that overlap evaluation topics. We run each content retriever to depth 1,000 on these queries and accumulate each document's mean reciprocal-rank contribution. This is a one-time offline cost of 2,000 queries per retriever. We then check that the estimate is stable by comparing tables built from disjoint query samples.

The analysis program has three parts.

First, we measure whether spurious consensus exists. For each pair of retrievers, we compute the correlation of their query-marginal tables. We then test whether plain RRF's top ten over-represents documents with high marginal pull, beyond what their relevance labels explain.

Second, we evaluate retrieval quality overall and, crucially, on relevant answers stratified by exposure.

Third, we test the speed side of the tradeoff. The prior ColBERT fusion system reported query latencies above 20 seconds (ARQMath 2022). If much of what the dense channel contributes is actually popularity pull, then correcting the cheap channels should recover much of its sensitivity. We therefore compare corrected fusion of the three cheap channels against uncorrected fusion that includes ColBERT, at their respective latencies. This secondary hypothesis could easily fail, and we will report it either way.

Baselines

The naive reference point is text BM25 alone, followed by plain RRF with $k=60$ over the same channels, using raw upvotes as the metadata channel. This is what a practitioner would deploy by default, and it is the $\beta=0$ special case of our method.

The strongest reported system on this task is the 2022 structural-plus-ColBERT pipeline, which linearly interpolates structural and ColBERT scores (fusion_alpha05). It reports nDCG' of 0.508 on ARQMath-3 Task 1, first place at CLEF 2022 (ARQMath 2022). nDCG' is nDCG computed after removing unjudged documents. We will reproduce it on our indices and report both our reproduction and the published 0.508. Beating it means exceeding the reproduced system's nDCG' on ARQMath-3 Task 1 without tuning on ARQMath-3 topics. It also means doing so with at least equal nDCG' on the low-exposure stratum.

The supervised metadata competitor is learning to rank over text BM25, formula BM25, upvotes, accepted flags and tag matches, as in Zhong et al.. The file gives no reported Task 1 number for that 2021 system; we found no reported result on Lacuna for ARQMath Task 1 answer retrieval with that system. We retrain it with ARQMath-2 topics as training data, with the same features plus a log-exposure feature. This gives it every chance to learn a popularity discount.

We also run weighted RRF in the style of MMMORRF, with channel weights grid-searched on ARQMath-2. All tuned systems get the same ARQMath-2 tuning topics and a comparable hyperparameter search budget. That means about 50 configurations each, including our $\beta$ and $k$.

Ablations isolate each ingredient. The first applies the correction only, keeping raw votes. The second uses exposure-normalized votes only, with no correction. The third uses the correction with $\beta$ fixed at 1 (fully label-free) versus $\beta$ tuned. The fourth uses a shuffled correction table, which keeps the table's distribution but assigns values to random documents; this rules out gains from generic score-shape changes.

Why not just…?

Why not just apply one popularity penalty to the fused list, MACR-style? This is the most serious threat. If the popularity effect were the same across retrievers, one penalty tuned on ARQMath-2 would match ours. Our argument is that the overcount scales with the number of biased channels that retrieve a document, and that number varies by query. A formula-heavy query may involve only the path channel, where popularity pull is weak. A prose query may involve three pull-heavy channels. We will stratify ARQMath-3 topics by how many channels return each relevant answer. If the global penalty matches us in every stratum, the per-retriever claim is wrong, and we will say so.

Why not just learn channel weights, or learning to rank with a popularity feature? Channel weights, as in weighted RRF, act on a whole retriever. They cannot demote one hub document while keeping the channel's other results. Learning to rank can in principle learn a negative popularity weight. But ARQMath provides only on the order of a hundred judged topics per year, and those judgments come from pooled runs of popularity-agnostic systems. So the supervision is thin and itself tilted toward head documents. The retrained learning-to-rank baseline, given a log-exposure feature, tests this directly.

Why not just normalize votes by exposure and leave the other channels alone? This fixes the one channel whose bias is obvious. It also may be most of the gain, because raw votes feed popularity into fusion explicitly. The second ablation measures exactly this. If it captures nearly all the improvement, the paper's contribution shrinks to a metadata-cleaning result. The per-retriever correction would only be justified by the consensus analysis showing measurable pull in the text and dense channels.

Proposed Experiments

We tune on ARQMath-2 topics and test on ARQMath-3 Task 1. We report nDCG', P'@10 and MAP'.

To measure popularity, we report the mean log-exposure of the top ten, in the spirit of the popularity index reported for CLRec. We also report nDCG' separately on relevant answers in the bottom and top exposure terciles, and median and 95th-percentile latency per query.

Pooled judgments penalize methods that surface unseen tail answers, so we report the fraction of unjudged documents in each system's top ten. We also commission a small blind relevance judgment of unjudged top-ten answers from our method and from plain RRF, on 30 topics.

As an optional transfer check, we would move to SemEval-2016 Task 3, with author reputation as the metadata channel. There the strongest reported result is MAP 0.8086 for UIA-LSTM-CNN (Hybrid Attentive Answer Selection). This check asks whether the correction helps even when the candidate pool per question is small.

Three results decide the proposal. First, the consensus analysis must show positive correlation of query-marginal pull across at least two content channels; if the channels are not jointly biased, the premise fails. Second, label-free corrected fusion ($\beta=1$) must raise low-exposure nDCG' over plain RRF without lowering high-exposure nDCG' by more than a small margin. The margin will be fixed before testing, at one nDCG' point. Third, the method must beat both the global post-hoc penalty and exposure-normalized votes alone, specifically on topics where relevant answers are retrieved by several popularity-biased channels. Winning only on the overall average would not establish the per-retriever mechanism.