Exploring “Just Noticeable” Group Fairness in Rankings

Mallak Alkhathlan, Hilson Shrestha, Lane Harrison, Elke A. RundensteinerAAAI 2025 · DOI

Abstract: The plethora of fairness metrics developed for ranking-based decision-making raises the question which metrics align best with people’s perceptions of fairness, and why? Most prior studies examining people’s perceptions of fairness metrics tend to use ordinal rating scales (e.g., Likert scales). However, such scales can be ambiguous in their interpretation across participants and offer imprecise connections to specific interface features. We address this gap by adapting two-alternative forced choice methodologies—used extensively outside the fairness community for comparing visual stimuli—to quantitatively compare participant perceptions, fairness metrics, and ranking characteristics. We report a crowdsourced experiment with 224 participants across four conditions: two popular rank fairness metrics—ARP and NDKL—and two ranking characteristics—lists of 20 and 100 candidates—resulting in over 170,000 individual judgments. Our quantitative results show systematic patterns of differences in the metrics, as well as surprising exceptions where fairness metrics disagree with people’s perceptions. Our qualitative analysis reveals an interplay between cognitive and visual strategies that affects people’s perceptions of fairness. From these results, we discuss future work in aligning fairness metrics with people’s perceptions, and highlight the need and benefits of expanding methodologies for fairness studies.

Introduction: The Gap Between Math and Perception

As AI-driven ranking systems become central to high-stakes decisions—such as hiring, college admissions, and lending—researchers have developed various mathematical metrics to measure "group fairness." However, a critical question remains: Do these mathematical definitions align with human intuition?

Most existing research relies on Likert scales (e.g., rating fairness from 1 to 7), which are often noisy and prone to bias. Because ranking is inherently complex—where the specific position of a candidate significantly impacts the perceived outcome—this study seeks a more rigorous, quantitative method to identify the threshold at which a human actually "notices" a change in fairness.

Methodology: The 2AFC Approach

To address the limitations of traditional rating scales, the researchers adapted the Two-Alternative Forced Choice (2AFC) method from psychophysics. In this setup, participants are not asked to rate a single list in isolation. Instead, they are shown two ranked lists side-by-side and must choose which one appears "more fair."

The study interface for the Two-Alternative Forced Choice (2AFC) task.

By aggregating thousands of these binary choices, the researchers calculated the Just Noticeable Difference (JND)—the minimum mathematical change in a fairness metric required for a person to reliably detect a difference. This methodology allows for a precise mapping of human perception onto mathematical scales.

Study Design and Metrics

The experiment involved 224 participants on Prolific, generating over 170,000 individual judgments. The study utilized a 2x2 design comparing two list sizes (20 and 100 candidates) and two prominent fairness metrics:

  1. Attribute Rank Parity (ARP): A metric that measures fairness based on the overall distribution of groups across the entire list.
  2. Normalized Discounted Cumulative KL-divergence (NDKL): A "top-heavy" metric that penalizes unfairness more severely if it occurs at the beginning of the ranking.
The complete study pipeline and the systematic range of fairness scores tested.

Rather than using static datasets, the researchers algorithmically generated synthetic rankings. This allowed them to systematically vary fairness levels from 0.0 (perfectly fair) to 1.0 (completely unfair), controlling the exact "Fair Distance" between stimuli.

Key Insights

The "Long List" Advantage

Counter-intuitively, participants were more precise and consistent when judging lists of 100 candidates compared to lists of 20. Longer lists likely provide more "visual evidence," making patterns of bias—such as clusters of a single group—easier to identify than in shorter, more sparse lists.

Metric Mismatch

The ARP metric (overall parity) consistently yielded lower JNDs than NDKL. This suggests that human intuition aligns more closely with global distribution rather than the strict "top-heavy" focus of NDKL. Humans appear to perceive fairness as a property of the entire list.

Stability at the Extremes

Perception is most acute at the edges of the scale. Participants were exceptionally good at detecting differences when a list was either perfectly fair or completely unfair. However, in the middle ranges (moderate unfairness), perception became significantly more "blurry" and inconsistent.

Cognitive vs. Visual Strategies

Qualitative analysis revealed that participants employed distinct strategies. Some were analytical, counting ratios or checking for 50/50 splits. Others were visual, looking for "color symmetry" or the "clumping" of group indicators (yellow vs. purple bars).

Results and Implications

In the ARP condition for 100-item lists, the JND at the "perfectly fair" level was only 0.10, meaning participants could distinguish a fair list from a slightly unfair one 98% of the time. Conversely, for the NDKL metric on 20-item lists, the JND was often double or triple that of the 100-item lists, highlighting how list size complicates fairness judgments.

These findings suggest that fairness-aware dashboards should be designed with human perception in mind—perhaps by highlighting unfairness only when it reaches a human-perceptible level, or by providing explanatory guidance when mathematical unfairness is present but visually subtle. Future work will look to expand these findings beyond binary groups to more complex, intersectional real-world scenarios.

Summary updated

Abstract

The plethora of fairness metrics developed for ranking-based decision-making raises the question which metrics align best with people’s perceptions of fairness, and why? Most prior studies examining people’s perceptions of fairness metrics tend to use ordinal rating scales (e.g., Likert scales). However, such scales can be ambiguous in their interpretation across participants and offer imprecise connections to specific interface features. We address this gap by adapting two-alternative forced choice methodologies—used extensively outside the fairness community for comparing visual stimuli—to quantitatively compare participant perceptions, fairness metrics, and ranking characteristics. We report a crowdsourced experiment with 224 participants across four conditions: two popular rank fairness metrics—ARP and NDKL—and two ranking characteristics—lists of 20 and 100 candidates—resulting in over 170,000 individual judgments. Our quantitative results show systematic patterns of differences in the metrics, as well as surprising exceptions where fairness metrics disagree with people’s perceptions. Our qualitative analysis reveals an interplay between cognitive and visual strategies that affects people’s perceptions of fairness. From these results, we discuss future work in aligning fairness metrics with people’s perceptions, and highlight the need and benefits of expanding methodologies for fairness studies.
BibTeX
@inproceedings{alkhathlan2025exploring,
  author = {Mallak Alkhathlan and Hilson Shrestha and Lane Harrison and Elke A. Rundensteiner},
  title = {{Exploring “Just Noticeable” Group Fairness in Rankings}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aies.v8i1.36532},
}
Figure