The disparate equilibria of algorithmic decision making when individuals invest rationally

Lydia T. Liu, Ashia Wilson, Nika Haghtalab, Adam Tauman Kalai, Christian Borgs, Jennifer T. Chayes

Abstract

The long-term impact of algorithmic decision making is shaped by the dynamics between the deployed decision rule and individuals' response. Focusing on settings where each individual desires a positive classification—including many important applications such as hiring and school admissions—we study a dynamic learning setting where individuals invest in a positive outcome based on their group's expected gain and the decision rule is updated to maximize institutional benefit. By characterizing the equilibria of these dynamics, we show that natural challenges to desirable long-term outcomes arise due to heterogeneity across groups and the lack of realizability. We consider two interventions: decoupling the decision rule by group and subsidizing the cost of investment. We show that decoupling achieves optimal outcomes in the realizable case but has discrepant effects that may depend on the initial conditions otherwise. In contrast, subsidizing the cost of investment is shown to create better equilibria for the disadvantaged group even in the absence of realizability.

Introduction

Most research in algorithmic fairness treats population data as a static snapshot. In reality, people respond strategically to the algorithms that judge them. For instance, if a hiring algorithm consistently fails to recognize qualified candidates from a specific demographic, individuals in that group have less incentive to invest the time and money required to become qualified in the first place. This creates a pernicious feedback loop: the algorithm's bias discourages investment, leading to lower qualification rates, which the algorithm then "correctly" identifies in the next round, reinforcing the original bias. This paper seeks to model these dynamics and evaluate whether common interventions—like using different thresholds for different groups or providing subsidies—actually lead to desirable long-term equilibria.

The Model

The authors propose a game-theoretic dynamical system model to capture the interaction between individuals and institutions. The model consists of three main components:

  1. Individual Response: Individuals decide whether to invest ($y$) in becoming "qualified" ($Y=1$) based on the expected reward (the probability of being accepted by the algorithm) minus the cost of investment.
  2. Institutional Response: The institution (e.g., a company or school) updates its decision rule (a classifier $\theta$) to maximize its own utility based on the current population distribution.
  3. Dynamics: This process repeats, moving toward an equilibrium where the population's qualification rate and the institution's decision rule are mutually best responses.
Causal model of individual investment and institutional observation.

The causal structure of this interaction is depicted in Figure 1. Here, $y$ represents the investment decision, which determines the true qualification state $Y$. The observed feature $X$, which the institution uses for classification, is influenced by both the qualification state $Y$ and the group membership $A$. The paper mathematically analyzes these equilibria under different conditions of realizability (whether the observable features can perfectly predict qualification) and heterogeneity (how much feature distributions differ between groups).

Experimental Setup

To test their theoretical findings, the authors use a semi-synthetic simulation based on the FICO credit score dataset (from a 2007 Federal Reserve report). This dataset contains aggregate statistics on credit scores across different racial groups (Asian, Black, Hispanic, White) and their subsequent repayment outcomes. By fitting Beta distributions to the empirical data, the authors simulate "non-realizable" settings where credit scores overlap significantly between those who repay and those who default. This allows them to test their theoretical findings on data that reflects real-world complexities where features are noisy indicators of true qualification.

Key Insights and Results

The analysis reveals several critical properties of the long-term equilibria:

  • Realizability as a Safety Net: If an algorithm can perfectly (or nearly perfectly) distinguish qualified from unqualified individuals using observable features, the system naturally converges to a high-qualification, high-utility equilibrium.
  • The Trap of Multiple Equilibria: In "non-realizable" settings (where features are noisy), a single group can have multiple stable equilibria—one where they invest heavily and one where they don't. Where they end up depends heavily on where they started.
  • The "Decoupling" Double-Edged Sword: Using group-specific classifiers (decoupling) is often proposed as a fairness intervention. The authors prove it is optimal if the problem is "realizable" for each group. However, if the features are noisy, decoupling can actually harm a disadvantaged group by trapping them in a low-investment equilibrium that they might have avoided under a joint rule. Simulation results using FICO data showed that decoupling could lead to a 3% to 25% drop in final qualification rates for certain groups.
  • Subsidies are More Robust: Unlike decoupling, subsidizing the cost of investment (e.g., lowering tuition or providing training) consistently shifts the equilibrium toward higher qualification rates for the disadvantaged group, regardless of whether the features are noisy or perfect.

Furthermore, the authors identify cases where the system never reaches a steady state but instead enters a "limit cycle," where the algorithm and the population's behavior oscillate indefinitely.

Discussion and Future Work

While the model provides deep insights into long-term fairness dynamics, it has limitations. It assumes the institution can see the true qualification of everyone it "hires," ignoring the "selective labels" problem where performance is only known for accepted candidates. Additionally, the model assumes individuals can always choose to invest, whereas real-world systemic barriers may prevent investment regardless of incentives. Future work should explore how finite sampling noise affects the stability of these equilibria and investigate more complex societal processes beyond simple rational investment.

Summary updated

Abstract

The long-term impact of algorithmic decision making is shaped by the dynamics between the deployed decision rule and individuals' response. Focusing on settings where each individual desires a positive classification---including many important applications such as hiring and school admissions, we study a dynamic learning setting where individuals invest in a positive outcome based on their group's expected gain and the decision rule is updated to maximize institutional benefit. By characterizing the equilibria of these dynamics, we show that natural challenges to desirable long-term outcomes arise due to heterogeneity across groups and the lack of realizability. We consider two interventions, decoupling the decision rule by group and subsidizing the cost of investment. We show that decoupling achieves optimal outcomes in the realizable case but has discrepant effects that may depend on the initial conditions otherwise. In contrast, subsidizing the cost of investment is shown to create better equilibria for the disadvantaged group even in the absence of realizability.
BibTeX
@inproceedings{liu2020disparate,
  author = {Lydia T. Liu and Ashia Wilson and Nika Haghtalab and Adam Tauman Kalai and Christian Borgs and Jennifer T. Chayes},
  title = {{The disparate equilibria of algorithmic decision making when individuals invest rationally}},
  booktitle = {FAT},
  year = {2020},
  doi = {10.1145/3351095.3372861},
}
Figure