Introduction
AI-driven Applicant Tracking Systems (ATS) are increasingly used to rank job candidates, but their "black box" nature obscures potential biases and unfairness. This paper outlines a framework for designing eXplainable AI (XAI) in algorithmic hiring by mapping out key stakeholders and gathering their specific needs through participatory design. Ultimately, it lays the groundwork for creating transparent, fair, and actionable AI ranking systems that serve both recruiters and applicants.
The hiring process is complex, socially sensitive, and increasingly automated. To handle massive pools of applicants, Human Resources (HR) departments rely on AI models to screen and rank CVs against job descriptions. However, these AI models are notoriously opaque. This lack of transparency hides potential biases—such as models learning to discriminate against women, minorities, or people with disabilities based on historical data. Because commercial ATS software is proprietary, auditing these systems is difficult. This paper addresses the urgent need for eXplainable AI (XAI) in hiring, focusing on who needs explanations and what information they require.
Methodology
Rather than proposing a new mathematical XAI algorithm, the authors take a software engineering and human-centered AI approach. The methodology consists of two main phases:
- Process Mapping and Stakeholder Analysis: The authors formally map the hiring process to identify everyone impacted by an AI-based ATS. They categorize stakeholders as internal (HR, IT, management) vs. external (candidates, auditors, regulators) and by their level of influence (primary, secondary, tertiary).
- Participatory Design: To elicit XAI requirements, the authors use a multi-stakeholder participatory design approach. This involves directly interviewing HR professionals to understand their workflows and planning Participatory Action Research (PAR) with diverse, legally protected groups of candidates (e.g., refugees, people with disabilities) to understand their needs for fairness and feedback.
Key Results: Stakeholder Mapping
A primary outcome of this preliminary research is the identification and mapping of 12 distinct stakeholder groups. These groups are categorized based on their relationship to the AI-based ATS and the impact the system has on their roles or lives.
Preliminary interviews with HR professionals revealed that while transparency is essential for trust, recruiters will "never follow AI blindly." They highly value comparative explanations, such as highlighting specific skill differences between top candidates.
Key Insights and Design Requirements
The research highlights that XAI is not a one-size-fits-all solution. Different stakeholders have vastly different informational needs:
- Candidates require actionable feedback. The most useful explanations are often actionable counterfactuals—explaining what realistic changes (e.g., acquiring a specific certification) would have moved them into the interview pool.
- HR Professionals need feature weights to trust the system and "what-if" analysis capabilities to understand how changing filters affects rankings.
- Auditors and Legal Teams require aggregate data to check for demographic biases and ensure compliance with anti-discrimination laws.
Furthermore, explanations can improve human decision-making. By revealing how AI scores are generated, HR can identify flaws in their own job descriptions and revise them to be fairer and more accurate.
Future Work
The immediate next step is executing the planned Participatory Action Research (PAR), involving 21 in-person meetings across 7 European cities with diverse candidate groups. Ultimately, the FINDHR project aims to use these elicited requirements to build and evaluate actual post-hoc and explainable-by-design XAI algorithms in real-world hiring scenarios.
Summary updated