How Fair is Fairness-aware Representative Ranking?

Akrati Saxena, George Fletcher, Mykola Pechenizkiy

TL;DR

This paper highlights a critical flaw in existing fairness-aware representative ranking (FRR) methods: they can inadvertently create unfairness when certain groups are less active on a platform. The authors propose new definitions for individual and group unfairness and outline methods to generate fair rankings even when universal representation ratios are unknown, offering a more robust approach to fair ranking.

What Problem Does This Paper Address?

Many online platforms, from job boards to search engines, rely on ranking algorithms to present information. The position of an item or candidate in a ranking significantly impacts their visibility and opportunities. While fairness-aware ranking methods aim to ensure equitable representation across different demographic groups (e.g., gender, ethnicity), a key challenge arises when the actual user base of a platform (set L) doesn't perfectly reflect the overall eligible population (universal set U). Specifically, if a particular group is "sub-active" on the platform (meaning fewer members of that group are present or active), standard FRR methods can lead to unfair outcomes. This is because FRR typically aims to maintain the representation ratio of groups as seen in the universal set, but if the platform's actual user base is skewed, this can disadvantage members of less active groups, even if they are highly qualified. The paper addresses this gap by investigating the sources of unfairness in FRR when group activity varies and proposing solutions to mitigate it.

Methods

The paper introduces new definitions for individual unfairness and group unfairness to better capture the nuances of fairness in sub-active scenarios.

  • Individual Unfairness: A candidate is individually unfair if they would have been ranked within the top-k in the universal set but are not in the platform's ranking, despite being present on the platform.
  • Group Unfairness: A group is unfairly treated if the number of its members in the top-k ranked results on the platform is less than what would be expected based on their proportion in the universal set.

The authors then propose two "ideal" solutions for generating fair rankings:

  • Ideal Individual Fair Representative Ranking (IFRR): This method simply filters the universal representative ranking (URR) to include only candidates present on the platform. It ensures that if a candidate is qualified and would have been ranked highly universally, they are considered if they are on the platform.
  • Ideal Group Fair Representative Ranking (GFRR): This method aims to maintain the group representation ratios from the universal set within the platform's ranking. It uses algorithms that prioritize candidates from underrepresented groups in the platform's top-k results, ensuring that group proportions are met as closely as possible.

The paper also discusses the challenges of generating these rankings when the universal representation ratio is unknown, suggesting estimation techniques like surveys or statistical methods as future work.

Datasets

The experimental results are presented using synthetic datasets. These datasets are designed to simulate scenarios with varying group activity levels and a protected attribute with three values (representing groups G1, G2, G3). This allows for controlled study of the impact of sub-activity on fairness. The authors note that a real-world example of such a situation would be gender classes with 'female', 'male', and 'unknown' values.

Key Insights and Ideas

  • Sub-activity is a critical source of unfairness: The core insight is that existing FRR methods, while aiming for fairness, can become unfair when certain demographic groups are less represented or active on a platform. This is a significant practical challenge for real-world applications.
  • Universal vs. Platform Representation: The distinction between the universal representation ratio and the platform's actual user base representation is crucial. Relying solely on the platform's data can mask underlying unfairness.
  • Individual vs. Group Fairness Interplay: The paper explores the relationship between individual and group fairness, showing that while individual fairness-aware solutions don't necessarily guarantee group fairness, group fairness-aware solutions tend to imply individual fairness.
  • The challenge of unknown universal ratios: A major practical hurdle is that platforms often don't know the true universal representation ratios. This necessitates methods for estimating these ratios, which is highlighted as a key area for future research.

Key Results

The paper demonstrates through simulations that:

  • As the "activeness" of a group (e.g., Gg) decreases on a platform, the number of candidates from that group who miss opportunities (i.e., are not ranked in the top-k) increases linearly.
  • The Average Rank Difference metric shows that standard LRR (Local Representative Ranking) performs poorly when a group's active fraction is low, indicating significant unfairness compared to the proposed IFRR (Individual Fair Representative Ranking).
  • The proposed GFRR (Group Fair Representative Ranking) method performs comparably to an "ideal" group fairness method, effectively maintaining representation ratios even with varying group activity.
  • The Skew Ratio metric also illustrates the effectiveness of GFRR in mitigating group unfairness, showing it performs as well as the ideal method.

Limitations and Future Work

  • Unknown Universal Representation Ratio: The paper acknowledges that in real-world scenarios, the universal representation ratio is often unknown. While they propose estimation methods (surveys, statistical techniques), these are not fully developed or evaluated within this paper and are left for future work.
  • IFRR Complexity: Generating an ideal IFRR is complex and highly dependent on the specific nature of missing data (i.e., which candidates are missing). The paper suggests that metadata about why candidates join or don't join a platform would be beneficial.
  • Platform-Specific Solutions: The proposed solutions for IFRR are based on the assumption of random missingness. Extending these to scenarios where candidate joining probability is correlated with their score or other characteristics (e.g., higher-ranked candidates are more likely to join) is identified as an open question.
  • Real-world Datasets: While synthetic datasets are used for controlled experiments, applying and validating these methods on diverse, real-world datasets with various protected attributes and platform dynamics remains a crucial next step.

Summary updated

BibTeX
@inproceedings{saxena2021how,
  author = {Akrati Saxena and George Fletcher and Mykola Pechenizkiy},
  title = {{How Fair is Fairness-aware Representative Ranking?}},
  booktitle = {The Web Conference},
  year = {2021},
  doi = {10.1145/3442442.3453458},
}
Figure