How Fair is Fairness-aware Representative Ranking and Methods for Fair Ranking

Akrati Saxena, George Fletcher, Mykola Pechenizkiy

TL;DR

This paper identifies a critical flaw in existing fairness-aware ranking systems: they can be unfair if certain demographic groups are less active on a platform. The authors propose new definitions for individual and group unfairness and outline methods to generate fair rankings even when universal demographic data is unknown, offering a more robust approach to equitable opportunity in ranked systems.

What Problem Does This Paper Address?

Many online platforms, from job boards to recommendation systems, rely on ranking algorithms to present users with relevant information. These rankings can significantly impact opportunities, as top positions often lead to more visibility and engagement. While fairness-aware ranking methods aim to mitigate biases, particularly against minority groups, this paper highlights a crucial oversight: these methods often assume that all eligible candidates are equally active or present on the platform. When a group is "sub-active" (i.e., fewer members participate or are visible), existing fairness-aware ranking algorithms can inadvertently create unfairness. This can lead to deserving candidates from less active groups being overlooked, even if they would have been fairly represented based on the overall population. The paper aims to address this by defining and quantifying this specific type of unfairness and proposing solutions that account for varying group activity levels.

Methods

The paper introduces new definitions for individual unfairness and group unfairness that specifically address the scenario of sub-active groups.

  • Individual Unfairness: A candidate is individually unfair if they would have been ranked within the top-k in the universal dataset but are not in the platform's ranked list, despite being eligible.
  • Group Unfairness: A group is unfairly treated if the number of its members in the platform's top-k results is less than what would be expected based on their representation in the universal dataset.

Based on these definitions, the authors propose two "ideal" methods for generating fair rankings:

  • Ideal Individual Fair Representative Ranking (IFRR): This method generates a ranking by simply filtering the universal ranking to include only candidates present on the platform.
  • Ideal Group Fair Representative Ranking (GFRR): This method aims to maintain the proportional representation of each group in the top-k results, mirroring the universal dataset's proportions, even when groups are sub-active.

The paper also outlines algorithms for generating these ideal rankings when the universal representation ratio is known. For the more challenging scenario where this ratio is unknown, the authors suggest estimating it through surveys or statistical methods, which would then allow the application of their proposed ideal ranking methods.

Datasets

The experimental results are presented using synthetic datasets. These datasets were created to specifically study the impact of varying group "activeness" (i.e., the proportion of members from a group present on the platform). The data features a protected attribute with three values (e.g., 'g', 'b', 'u'), categorizing candidates into three groups. The authors assume one group is less active, with its active member ratio ranging from 0.1 to 0.9, while other groups are fully active. This controlled environment allows for a clear analysis of how sub-activeness affects fairness.

Key Insights and Ideas

  • Sub-activeness is a critical fairness blind spot: Existing fairness-aware ranking methods, while well-intentioned, can fail when demographic groups have different participation rates on a platform. This is a significant practical challenge.
  • Individual and Group Fairness are Intertwined but Distinct: The paper formally defines how individual unfairness can lead to group unfairness, and vice-versa, providing a clearer framework for analyzing these concepts in the context of sub-active groups.
  • Ideal vs. Practical Solutions: The distinction between "ideal" rankings (assuming full knowledge) and practical solutions (dealing with missing or incomplete data) is crucial. The paper lays the groundwork for both, highlighting the challenges of the latter.
  • The Importance of Universal Representation: The core of the problem lies in the discrepancy between the universal demographic distribution and the platform's observed distribution. Knowing or estimating the former is key to achieving true fairness.

Key Results

The paper's key results are primarily demonstrated through simulations and theoretical arguments:

  • Quantified Unfairness: Figure 2 visually demonstrates that as a group becomes less active (lower active fraction) and the number of top-k results increases, the number of candidates from that group who miss opportunities increases linearly.
  • IFRR vs. LRR Performance: Figure 3 shows that the proposed IFRR method significantly reduces average rank difference compared to the standard LRR (Local Representative Ranking) when a group's activeness decreases. This indicates IFRR is more robust to sub-activeness.
  • GFRR Performance: Figure 4 indicates that the proposed GFRR method performs comparably to an ideal group fairness method, effectively maintaining the desired representation ratio even with varying group activeness.

Limitations and Future Work

  • Unknown Universal Representation Ratio: The paper acknowledges that in real-world scenarios, the universal representation ratio is often unknown. While they propose estimation methods (surveys, statistical techniques), developing robust and efficient estimation strategies is identified as a key area for future work.
  • Platform-Specific Solutions: The proposed IFRR solution assumes candidates join uniformly at random. The authors note that in reality, a candidate's decision to join a platform might be correlated with their score or other characteristics. Developing IFRR solutions that account for these correlations (e.g., higher-ranked candidates being more likely to join) is an open question.
  • Generalizing to Complex Scenarios: The paper focuses on a single protected attribute. Extending the methods to handle multiple protected attributes and their complex interactions is a natural next step.
  • Trade-offs: While the paper proposes methods for fairness, the inherent trade-offs between fairness and other ranking objectives (like accuracy or relevance) are not deeply explored, which is a common challenge in this field.

Summary updated

Abstract

Rankings of people and items has been highly used in selection-making, match-making, and recommendation algorithms that have been deployed on ranging of platforms from employment websites to searching tools. The ranking position of a candidate affects the amount of opportunities received by the ranked candidate. It has been observed in several works that the ranking of candidates based on their score can be biased for candidates belonging to the minority community. In recent works, the fairness-aware representative ranking was proposed for computing fairness-aware re-ranking of results. The proposed algorithm achieves the desired distribution of top-ranked results with respect to one or more protected attributes. In this work, we highlight the bias in fairness-aware representative ranking for an individual as well as for a group if the group is sub-active on the platform. We define individual unfairness and group unfairness and propose methods to generate ideal individual and group fair representative ranking if the universal representation ratio is known or unknown. The simulation results show the quantified analysis of fairness in the proposed solutions. The paper is concluded with open challenges and further directions.
BibTeX
@inproceedings{saxena2021how,
  author = {Akrati Saxena and George Fletcher and Mykola Pechenizkiy},
  title = {{How Fair is Fairness-aware Representative Ranking and Methods for Fair Ranking}},
  booktitle = {The Web Conference},
  year = {2021},
  doi = {10.1145/3442442.3453458},
}
Figure