Discretionary Trees: Understanding Street-Level Bureaucracy via Machine Learning

Gaurab Pokharel, Sanmay Das, Patrick J. FowlerAAAI 2024 · DOI

TL;DR

This paper uses machine learning to analyze how "street-level bureaucrats" (e.g., social workers) make decisions about allocating homelessness services. They find that these decisions are surprisingly predictable, with simple rules explaining a significant portion, but that discretion is applied in a nuanced way that appears to benefit vulnerable populations.

What Problem Does This Paper Address?

Street-level bureaucrats, like social workers and police officers, are on the front lines of public service, making critical decisions about resource allocation. While they are tasked with implementing agency policies, they also have significant discretion. This discretion can be beneficial, allowing for individual circumstances to be considered, but it also opens the door to potential biases and inequities. Understanding how and why these bureaucrats exercise their discretion is crucial for ensuring fairness and improving the effectiveness of public services, especially in high-stakes areas like homelessness intervention. Existing research has largely focused on the outcomes of these decisions or the development of algorithmic decision-support tools, but this paper aims to directly model and understand the human decision-making process itself.

Methods

The authors employ machine learning techniques to model caseworker decision-making. Their core approach involves using short decision trees (SER-DT) to capture "simple heuristic rules" that street-level bureaucrats might use to make quick decisions in straightforward cases. This is contrasted with a more complex XGBoost model to understand decisions that deviate from these simple rules, which are interpreted as the application of caseworker discretion. They also use Bayesian Additive Regression Trees (BART) to estimate the counterfactual marginal benefit of different interventions.

Datasets

The study utilizes administrative records from the homeless management information system (HMIS) in St. Louis, MO, collected between 2007 and 2014. This dataset includes household-level characteristics and records of interventions assigned (e.g., Emergency Shelter, Transitional Housing). The data was collected during a period when the allocation process was less formulaic, making it suitable for studying human discretion.

Key Insights and Ideas

  • Predictable, Yet Nuanced Decisions: Caseworker decisions are surprisingly predictable, suggesting that simple heuristics play a significant role. However, the application of discretion is not random; it's applied strategically.
  • Discretion Favors Lower Vulnerability: Caseworkers tend to apply discretion primarily to households that are considered less vulnerable, rather than those with the most acute needs.
  • Marginal Benefit Maximization: When discretion is used to assign individuals to more intensive interventions (like Transitional Housing), the expected marginal benefit to those individuals is significantly higher than if they were assigned randomly. Conversely, there's no significant loss in marginal benefit when discretion leads to less intensive interventions.
  • Human Expertise Aligns with ML: Caseworkers, without access to explicit outcome predictions, seem to align their discretionary decisions with what machine learning models predict as beneficial, suggesting a sophisticated understanding of the factors influencing outcomes.

Key Results

  • Simple Decision Rules: Short decision trees (SER-DT) achieved Area Under the ROC Curve (AUC) scores of 0.76 for Emergency Shelter (ES) and 0.87 for Transitional Housing (TH), indicating that simple rules can capture a substantial portion of these decisions.
  • High Consistency: More complex models (XGBoost) achieved AUCs of 0.95 for ES and 0.94 for TH, demonstrating a high degree of consistency in caseworker decision-making, even beyond simple rules.
  • Vulnerability Analysis: Discretionary assignments (cases where the model's prediction differed from the actual assignment) were significantly associated with lower vulnerability scores compared to the predicted population.
  • Marginal Benefit: The EStoTH (predicted ES, actual TH) group showed a significant marginal benefit, while the THtoES (predicted TH, actual ES) group did not show a corresponding significant loss.

Limitations and Future Work

  • Data Limitations: The study uses data from before 2014, a period before more standardized prioritization tools like VI-SPDAT were universally adopted. While this allows for the study of discretion, it might not fully reflect current practices. The paper acknowledges that VI-SPDAT itself has faced criticism for potential biases.
  • Causality vs. Correlation: While the study shows strong correlations between caseworker decisions and outcomes, establishing definitive causality is complex.
  • Future Work: The authors suggest that their findings can inform the development of AI tools to better assist human experts in decision-making, focusing on areas where human judgment is most valuable and where AI can provide support without undermining procedural justice. They also highlight the need to understand the "bureaucratic counterfactual" – how human judgment interacts with automated systems.

Summary updated

BibTeX
@inproceedings{pokharel2024discretionary,
  author = {Gaurab Pokharel and Sanmay Das and Patrick J. Fowler},
  title = {{Discretionary Trees: Understanding Street-Level Bureaucracy via Machine Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i20.30236},
}
Figure