Dissatisfaction Induced by Pairwise Swaps

Alessandro Fabris, Gianmaria Silvello, Gian Antonio Susto, Asia J. BiegaIIR 2023 · OpenReview

Abstract

This paper introduces Dissatisfaction Induced by Pairwise Swaps (DIPS), a human-centric metric for evaluating pairwise fairness in ranking systems. By reframing algorithmic ranking errors as a measure of content producer dissatisfaction, DIPS addresses critical gaps in existing metrics, such as the lack of normative grounding, the failure to account for top-heavy browsing behavior, and the inadequate handling of ties in relevance scores. Through evaluations on synthetic and real-world datasets, the authors demonstrate that DIPS provides a more realistic assessment of fairness that aligns with user exposure while offering unique insights for system interventions.

Introduction

In Information Access Systems (IAS)—including search engines and recommendation platforms—fairness is a multi-stakeholder concern. While much attention is paid to consumers, the producers of content are equally vital. When a platform ranks a lower-quality item above a higher-quality one, it creates a sense of being "cheated" for the slighted producer. Traditional metrics for measuring these errors, such as the Rank Efficiency Error (REE), have historically lacked a clear behavioral or philosophical justification. Furthermore, they often treat every rank position equally, ignoring the reality that visibility (and thus the impact of a swap) decays rapidly as one moves down a list.

Methodology

The authors bridge the gap between abstract mathematical metrics and human behavior by redefining pairwise fairness through the lens of producer sentiment. The methodology consists of three primary components:

  1. Browsing Model Connection: The researchers mathematically prove that standard pairwise metrics are variants of Kendall’s Tau, which implicitly assumes a "trivial" browsing model where users are equally likely to see every item. DIPS replaces this with a realistic model where top ranks carry more weight.
  2. Defining Dissatisfaction: A "discordant pair"—where a less relevant item is ranked higher than a more relevant one—is formally interpreted as a source of producer dissatisfaction.
  3. The DIPS Metric: The proposed metric incorporates a weighting function $F(k)$ (e.g., logarithmic or exponential decay) to reflect that swaps at the top of the list cause significantly more dissatisfaction. Additionally, it introduces a parameter ($c_t$) to penalize "partial" unfairness when a system arbitrarily favors one group over another among items with identical relevance scores.

Experimental Evaluation

The paper evaluates DIPS using both synthetic and real-world datasets. Synthetic data allows for controlled testing of how the metric reacts to specific ranking distortions, while real-world data (from the WSDM '23 study) compares DIPS against traditional exposure-based measures on actual ranking distributions.

Comparison of DIPS and REE metrics across synthetic and real-world datasets.

Key Results and Insights

Experiments reveal that DIPS behaves similarly to exposure-based metrics like Equity of Attention (EA) because both prioritize the top of the ranking. This makes DIPS a more practical tool for real-world auditing than older, "flat" pairwise metrics like REE.

Crucially, DIPS captures a different construct of fairness than exposure-based metrics. This distinction allows for "fairness interventions" where a system can improve group equity—giving a marginalized group more visibility—without significantly increasing the dissatisfaction of the privileged group. Furthermore, by addressing ties, DIPS closes a common loophole where algorithms might systematically favor one group's items over another's when relevance scores are equal (e.g., both having 4-star ratings).

Conclusion and Future Work

By mapping algorithmic errors to "dissatisfaction," the authors shift the focus from abstract error rates to platform loyalty and producer retention. While DIPS provides a robust framework, the authors acknowledge that human psychology regarding dissatisfaction (such as loss aversion) may be more complex than a linear visibility model. Future work will involve tuning parameters like the tie-handling constant ($c_t$) for specific platform contexts and investigating how perceived injustices influence long-term producer behavior and overall platform health.

Summary updated

BibTeX
@inproceedings{fabris2023dissatisfaction,
  author = {Alessandro Fabris and Gianmaria Silvello and Gian Antonio Susto and Asia J. Biega},
  title = {{Dissatisfaction Induced by Pairwise Swaps}},
  booktitle = {IIR},
  year = {2023},
  url = {https://openreview.net/forum?id=m2RuEMmWcI},
}
Figure