The Initial Screening Order Problem

José M. Álvarez, Antonio Mastropietro, Salvatore RuggieriWSDM 2023 · DOI · OpenReview

Abstract We investigate the role of the initial screening order (ISO) in candidate screening tasks, such as employee hiring and academic admissions, in which a screener is tasked with selecting $k$ candidates from a candidate pool. The ISO refers to the order in which the screener searches the candidate pool. Today, it is common for the ISO to be the product of an information access system, such as an online platform or a database query. The ISO has been largely overlooked in the literature, despite its potential impact on the optimality and fairness of the chosen $k$ candidates, especially under a human screener. We define two problem formulations describing the search behavior of the screener under the ISO: the best-$k$, where the screener selects the $k$ best candidates; and the good-$k$, where the screener selects the $k$ first good-enough candidates. To study the impact of the ISO, we introduce a human-like screener and compare it to its algorithmic counterpart, where the human-like screener is conceived to be inconsistent over time due to fatigue. In particular, our analysis shows that the ISO, under a human-like screener solving for the good-$k$ problem, hinders individual fairness despite meeting group level fairness, and hampers the optimality of the selected $k$ candidates. This is due to position bias, where a candidate's evaluation is affected by its position within the ISO. We report extensive simulated experiments exploring the parameters of the best-$k$ and good-$k$ problems for the algorithmic and human-like screeners.

Introduction

In modern recruitment, candidates are rarely evaluated in a vacuum. Instead, they are presented to screeners through Information Access Systems (IAS) in a specific sequence known as the Initial Screening Order (ISO). While significant research has focused on the fairness of ranking algorithms, the interaction between the ISO and human cognitive limitations—specifically fatigue—remains poorly understood. This paper addresses this gap by examining how the ISO influences the optimality and fairness of selection outcomes when a human-like screener is involved.

Search Behaviors and the Human-Like Screener

We formalize two distinct search strategies:

  1. Best-$k$: The screener evaluates the entire candidate pool and selects the top $k$ individuals.
  2. Good-$k$: The screener performs a partial search, stopping as soon as they find the first $k$ candidates who meet a quality threshold ($\psi$).

To model realistic human behavior, we introduce a Human-Like Screener model. Unlike a consistent algorithm, this screener accumulates fatigue ($\Phi$) over time, which is modeled as noise added to a candidate's true score. We consider two types of noise: Inconsistency ($\epsilon_1$), where score variance increases (the "sloppy" recruiter), and Negative Bias ($\epsilon_2$), where the mean score decreases as the screener tires (the "grumpy" recruiter).

Simulation Framework

The study utilizes Monte Carlo simulations to evaluate these models across diverse scenarios. Candidate pools are generated using various score distributions (Symmetric, Asymmetric, and Increasing) to represent different talent densities. The ISO is varied from random shuffles to pre-ranked lists where the order is correlated with candidate quality ($\rho$).

Figure 1: Candidate score distributions and the resulting Ratio to Baseline (RtB) for different pool sizes (n) and selection targets (k).

Key Results and Discussion

Optimality Loss and Position Bias

Our simulations demonstrate that in the "Good-$k$" scenario with a random ISO, screeners frequently miss the most qualified candidates. The Ratio to Baseline (RtB)—a measure of utility relative to an optimal search—decreases significantly as the screening threshold ($\psi$) increases. This indicates that as standards rise, the specific order of candidates becomes a dominant factor in selection.

This phenomenon introduces "position bias," which constitutes a violation of individual fairness. Two identical candidates are treated differently based solely on their position in the ISO. Interestingly, our results show that human-like screeners can often meet group-level fairness quotas while still failing individual fairness by selecting the "wrong" individuals from a protected group due to fatigue-induced errors.

Figure 3: Comprehensive simulation results illustrating the impact of score distributions, pool parameters, and ISO correlation on optimality (RtB) and selection similarity (Jaccard).

The Value of Pre-Ranking

While AI-driven ranking is often scrutinized for bias, our findings suggest it can mitigate the negative effects of human fatigue. By increasing the correlation ($\rho$) between the ISO and candidate quality, high-quality candidates appear earlier in the list. This reduces the "wasted effort" of the screener and ensures that the best candidates are evaluated before fatigue significantly impairs judgment. As seen in the simulation results, pre-ranking ensures the RtB remains high even when the screener stops early.

Conclusion

The ISO is a critical but often overlooked component of the candidate screening process. This work highlights how the combination of search order and human fatigue can lead to sub-optimal and unfair outcomes. Future research should investigate more complex fatigue models and the potential for dynamic thresholding, where screeners adjust their standards based on early observations in the candidate pool.

Summary updated

Abstract

We investigate the role of the initial screening order (ISO) in candidate screening tasks, such as employee hiring and academic admissions, in which a screener is tasked with selecting $k$ candidates from a candidate pool. The ISO refers to the order in which the screener searches the candidate pool. Today, it is common for the ISO to be the product of an information access system, such as an online platform or a database query. The ISO has been largely overlooked in the literature, despite its potential impact on the optimality and fairness of the chosen $k$ candidates, especially under a human screener. We define two problem formulations describing the search behavior of the screener under the ISO: the best-$k$, where the screener selects the $k$ best candidates; and the good-$k$, where the screener selects the $k$ first good-enough candidates. To study the impact of the ISO, we introduce a human-like screener and compare it to its algorithmic counterpart, where the human-like screener is conceived to be inconsistent over time due to fatigue. In particular, our analysis shows that the ISO, under a human-like screener solving for the good-$k$ problem, hinders individual fairness despite meeting group level fairness, and hampers the optimality of the selected $k$ candidates. This is due to position bias, where a candidate's evaluation is affected by its position within the ISO. We report extensive simulated experiments exploring the parameters of the best-$k$ and good-$k$ problems for the algorithmic and human-like screeners. The simulation framework is flexible enough to account for multiple screening settings, being an alternative to running real-world candidate screening procedures. This work is motivated by a real-world candidate screening problem studied in collaboration with an European company.
BibTeX
@inproceedings{alvarez2023initial,
  author = {José M. Álvarez and Antonio Mastropietro and Salvatore Ruggieri},
  title = {{The Initial Screening Order Problem}},
  booktitle = {WSDM},
  year = {2023},
  doi = {10.1145/3701551.3703497},
}
Figure