Abstract
This paper introduces Dissatisfaction Induced by Pairwise Swaps (DIPS), a human-centric metric for evaluating pairwise fairness in ranking systems. By reframing algorithmic ranking errors as a measure of content producer dissatisfaction, DIPS addresses critical gaps in existing metrics, such as the lack of normative grounding, the failure to account for top-heavy browsing behavior, and the inadequate handling of ties in relevance scores. Through evaluations on synthetic and real-world datasets, the authors demonstrate that DIPS provides a more realistic assessment of fairness that aligns with user exposure while offering unique insights for system interventions.
Introduction
In Information Access Systems (IAS)—including search engines and recommendation platforms—fairness is a multi-stakeholder concern. While much attention is paid to consumers, the producers of content are equally vital. When a platform ranks a lower-quality item above a higher-quality one, it creates a sense of being "cheated" for the slighted producer. Traditional metrics for measuring these errors, such as the Rank Efficiency Error (REE), have historically lacked a clear behavioral or philosophical justification. Furthermore, they often treat every rank position equally, ignoring the reality that visibility (and thus the impact of a swap) decays rapidly as one moves down a list.
Methodology
The authors bridge the gap between abstract mathematical metrics and human behavior by redefining pairwise fairness through the lens of producer sentiment. The methodology consists of three primary components:
- Browsing Model Connection: The researchers mathematically prove that standard pairwise metrics are variants of Kendall’s Tau, which implicitly assumes a "trivial" browsing model where users are equally likely to see every item. DIPS replaces this with a realistic model where top ranks carry more weight.
- Defining Dissatisfaction: A "discordant pair"—where a less relevant item is ranked higher than a more relevant one—is formally interpreted as a source of producer dissatisfaction.
- The DIPS Metric: The proposed metric incorporates a weighting function $F(k)$ (e.g., logarithmic or exponential decay) to reflect that swaps at the top of the list cause significantly more dissatisfaction. Additionally, it introduces a parameter ($c_t$) to penalize "partial" unfairness when a system arbitrarily favors one group over another among items with identical relevance scores.
Experimental Evaluation
The paper evaluates DIPS using both synthetic and real-world datasets. Synthetic data allows for controlled testing of how the metric reacts to specific ranking distortions, while real-world data (from the WSDM '23 study) compares DIPS against traditional exposure-based measures on actual ranking distributions.
Key Results and Insights
Experiments reveal that DIPS behaves similarly to exposure-based metrics like Equity of Attention (EA) because both prioritize the top of the ranking. This makes DIPS a more practical tool for real-world auditing than older, "flat" pairwise metrics like REE.
Crucially, DIPS captures a different construct of fairness than exposure-based metrics. This distinction allows for "fairness interventions" where a system can improve group equity—giving a marginalized group more visibility—without significantly increasing the dissatisfaction of the privileged group. Furthermore, by addressing ties, DIPS closes a common loophole where algorithms might systematically favor one group's items over another's when relevance scores are equal (e.g., both having 4-star ratings).
Conclusion and Future Work
By mapping algorithmic errors to "dissatisfaction," the authors shift the focus from abstract error rates to platform loyalty and producer retention. While DIPS provides a robust framework, the authors acknowledge that human psychology regarding dissatisfaction (such as loss aversion) may be more complex than a linear visibility model. Future work will involve tuning parameters like the tie-handling constant ($c_t$) for specific platform contexts and investigating how perceived injustices influence long-term producer behavior and overall platform health.
Summary updated