Impact of Bias on School Admissions and Targeted Interventions

Yuri Faenza, Swati Gupta, Xuan Zhang

TL;DR

This paper tackles the "filtering effect" in New York City's elite public high school admissions, where socioeconomic disparities lead to unequal opportunities. The authors propose and analyze targeted interventions, like vouchers for tutoring, to mitigate this bias and improve fairness, demonstrating their effectiveness through theoretical models and real-world data.

What Problem Does This Paper Address?

The paper addresses a significant issue in public school admissions: the "filtering effect." In New York City, specialized high schools (SHSs) rely heavily on the Specialized High School Admissions Test (SHSAT). However, SHSAT scores are influenced by socioeconomic status and access to test preparation, creating a disadvantage for students from lower-income backgrounds. This leads to a situation where a disproportionately small number of middle schools produce the majority of SHS admits, exacerbating segregation and limiting opportunities. While previous attempts to address this involved policy changes like quotas, they faced legal challenges and mixed results. This paper seeks to find a more effective and fair approach by focusing on pre-admission interventions.

Methods

The authors employ an "operations research" approach, deviating from traditional policy-focused methods. They develop a mathematical model of a continuous matching market with a "bias" factor. This bias represents the difference between a student's true potential and their perceived potential (measured by SHSAT scores), with disadvantaged students assumed to have their potential underestimated. They then analyze the impact of this bias on student assignments and introduce "vouchers" (representing resources like tutoring or scholarships) as interventions to "debias" students. The core of their methodology involves:

  • Modeling Bias: A multiplicative bias factor ($\beta$) is introduced, where a disadvantaged student's perceived potential is $\beta$ times their true potential.
  • Matching Algorithm: They use a stable matching framework, adapted for continuous markets, to model how students are assigned to schools based on perceived potential.
  • Intervention Analysis: They analyze the impact of deterministic and randomized voucher allocation policies on measures of "mistreatment" (how much worse a student is assigned due to bias).
  • Fairness Metrics: They focus on minimizing maximum mistreatment and the total mistreatment (Positive Area Under the Curve - PAUC).
  • Empirical Validation: They use real SHSAT score data from NYC to estimate model parameters and evaluate the proposed interventions.

Datasets

The primary dataset used is SHSAT admissions data from New York City for the 2016-17 academic year. This dataset includes SHSAT scores, student preference lists for SHSs, and information on whether students are classified as "disadvantaged" by the NYC Department of Education.

Key Insights and Ideas

  • Asymmetric Impact of Bias: The bias disproportionately harms disadvantaged students, with only a slight benefit to advantaged students.
  • Targeting Average Performers: Interventions (vouchers) are most effective when targeted at average-performing disadvantaged students, not necessarily the top performers. This is because a small improvement for an average student can lead to a larger jump in school assignment compared to a top performer.
  • Limitations of Deterministic Interventions: Deterministic voucher allocation can create incentives for students to underperform to qualify for the intervention.
  • Randomization for Fairness: Randomized voucher allocation policies can be incentive-compatible and individually fair, while still effectively reducing mistreatment.
  • Robustness to Model Misspecification: The core findings of the paper remain largely consistent even when the assumed bias model is slightly misspecified, suggesting the practical applicability of the proposed interventions.

Key Results

  • Optimal Intervention Target: Both theoretical analysis and empirical validation suggest that vouchers should be targeted at average-performing disadvantaged students to maximize fairness gains.
  • PAUC Reduction: The proposed interventions can significantly reduce aggregate mistreatment (PAUC) for disadvantaged students. For example, with a low budget (ĉ=0.1), the theoretical intervention achieved a PAUC reduction of up to 17.28%, with minimal difference compared to empirically optimal intervals.
  • Robustness: Even with various forms of bias (additive, uniform noise, Gaussian noise), the theoretical model's insights on targeting average performers hold, with differences in PAUC reduction generally being small, especially for lower budgets.
  • Incentive Compatibility: Deterministic policies struggle with incentive compatibility, but randomized policies (like Proportional-to-Mistreatment) can achieve this property.

Limitations and Future Work

  • Ex-Post Fairness Perception: Randomized interventions might still lead to perceptions of unfairness ex-post. The authors suggest future work could explore strategies like implementing voucher programs in carefully selected schools with low economic indices.
  • Definition of Disadvantaged Students: The analysis relies on the NYC DOE's definition of disadvantaged students. Changes to this definition would require recalibrating the model.
  • Model Assumptions: The paper assumes a multiplicative bias and that vouchers fully debias students. Relaxing these assumptions (e.g., stochastic-effect vouchers, more general bias models) is explored but leaves room for further theoretical analysis.
  • Heterogeneous Preferences: While the empirical validation uses heterogeneous preferences, the theoretical model assumes homogeneous preferences for tractability. Extending the theoretical framework to fully account for heterogeneous preferences is a direction for future work.
  • Potential Distribution: The Pareto distribution is used for student potentials. Investigating the impact of other distributions on the findings is also suggested.

Summary updated

Abstract

Traditionally, New York City's top 8 public schools select candidates solely based on their scores in the Specialized High School Admissions Test (SHSAT). These scores are known to be impacted by socioeconomic status of students and test preparation received in "feeder" middle schools, leading to a massive filtering effect in the education pipeline. For instance, around 50% (resp. 80%) of the students admitted to the top public high schools in New York City come from only 5% (resp. 15%) of the middle schools. The classical mechanisms for assigning students to schools do not naturally address problems like school segregation and class diversity, which have worsened over the years. The scientific community, including policy makers, have reacted by incorporating group-specific quotas, proportionality constraints, or summer school opportunities, but there is evidence that these can end up hurting minorities or even create legal challenges. We take a completely different approach to reduce this filtering effect of feeder middle schools, with the goal of increasing the opportunity of students with high economic needs. We model a two-sided market where a candidate may not be perceived at their true potential, and is therefore assigned to a lower level than the one (s)he deserves. We propose and study the effect of interventions such as additional training and scholarships towards disadvantaged students, and challenge existing mechanisms for scholarships. We validate these findings using SAT scores data from New York City high schools. We further show that our qualitative takeaways remain the same even when some of the modeling assumptions are relaxed.
BibTeX
@misc{faenza2020impact,
  author = {Yuri Faenza and Swati Gupta and Xuan Zhang},
  title = {{Impact of Bias on School Admissions and Targeted Interventions}},
  year = {2020},
  eprint = {2004.10846},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.10846},
}
Figure