Impact Evaluation in Matching Markets with General Tie-Breaking

Atila Abdulkadiroğlu, Joshua D. Angrist, Yusuke Narita, Parag A. Pathak2017 · DOI

Abstract

Many centralized matching schemes incorporate a mix of random lottery and non-lottery tiebreaking. A leading example is the New York City public school district, which uses criteria like test scores and interviews to generate applicant rankings for some schools, combined with lottery tie-breaking at other schools. We develop methods that identify causal effects of assignment in such settings. Our approach generalizes the standard regression discontinuity design to allow for many running variables and treatments, some of which are randomly assigned. We show that lottery variation generates assignment risk at non-lottery programs for applicants away from nonlottery cutoffs, while non-lottery variation randomizes applicants near cutoffs regardless of lottery risk. These methods are applied to evaluate New York City's school progress assessments, which give schools letter grades as a summary measure of quality. Our estimates reveal that although Grade A schools boost achievement, these gains emerge only for students who attend lottery schools. Attendance at a coveted Grade A screened school, including some of the highest performing in the district, generates no measurable effects. Evaluation methods that fail to take advantage of both lottery and non-lottery variation miss this difference in impact.

Introduction: The Challenge of Mixed Matching Markets

In many large urban school districts, students are assigned to schools via centralized algorithms like Deferred Acceptance. For researchers, these systems are potentially powerful natural experiments because they use tie-breakers to allocate scarce seats. When the tie-breaker is a lottery, it functions like a randomized controlled trial. When it is a test score, researchers can use Regression Discontinuity (RD) designs.

However, real-world markets like New York City (NYC) are often a complex mix. Some schools use lotteries, while others "screen" students based on grades or interviews. Standard methods struggle in these environments because a student’s risk of being assigned to a specific school depends on their preferences and their standing across multiple different types of tie-breakers simultaneously. This paper introduces a unified framework to account for both lottery and non-lottery variation, producing unbiased estimates of school quality.

Methodology: The Local Propensity Score

The authors develop a Local Propensity Score for matching markets with general tie-breaking, bridging the gap between lottery-based research and traditional RD designs.

Key components of the methodology include:

  1. Generalizing the DA Propensity Score: The framework extends previous work on lottery-based "Deferred Acceptance" (DA) propensity scores to include non-random running variables like exam scores.
  2. Local-to-Cutoff Logic: Near a screened school's admission cutoff, the probability of assignment converges to 0.5, effectively creating a local experiment similar to traditional RD.
  3. Exploiting "Disqualification" Risk: A critical insight is that lottery variation at a high-ranked school creates "assignment risk" at a lower-ranked screened school. If a student loses a lottery at their first choice, they are "randomly" sent down their list to a screened school, even if they are not near that screened school's specific merit cutoff.
  4. Large Market Approximation: To handle thousands of students and hundreds of schools, the authors use a continuum model where cutoffs are treated as fixed, providing a consistent estimator for real-world finite markets.

Data and Empirical Setting

The study utilizes comprehensive data from the New York City Department of Education (DOE) for the 2012-13 and 2013-14 school years. This includes student rank-order lists, school capacities, and priority groups. Crucially, the researchers accessed both the randomly assigned lottery numbers and the school-specific rankings for "screened" programs based on middle school GPA and test scores. These were linked to outcome data including standardized Regents exam scores, SAT scores, and high school graduation rates.

Key Findings: The "Elite Illusion"

The application of this new methodology reveals a phenomenon the authors call the "Elite Illusion." While raw data (OLS) suggested that attending a highly-rated "Grade A" school increased graduation rates by 13 percentage points, the causal (2SLS) estimate showed zero actual impact on graduation.

Further insights include:

  • Divergent Impacts: While "Grade A" schools provide a modest boost to English achievement (~0.03 standard deviations), this gain is driven entirely by lottery-based schools.
  • Screened School Performance: Attending a "Grade A" screened school—often the most prestigious and coveted in the district—resulted in no measurable improvement in test scores or graduation rates compared to attending a lower-ranked school. These schools appear to produce better outcomes primarily because they enroll students who were already high achievers.
  • Precision Gains: By combining lottery and non-lottery variation, the authors significantly increased the effective sample size. The "general risk" estimator was far more precise than using lottery risk alone, reducing standard errors by up to 50% for certain outcomes.

Conclusion and Future Applications

This research demonstrates that failing to account for the interplay between lottery and non-lottery variation can lead to significant misinterpretations of school quality. The framework developed here is not limited to education; it can be applied to other complex matching markets, such as the U.S. National Residency Matching Program for medical graduates or auction markets where bids act as non-random tie-breakers. Future work may further refine these models to account for imperfectly recorded tie-breakers and clerical errors in large-scale administrative datasets.

Summary updated

Abstract

Many centralized matching schemes incorporate a mix of random lottery and non-lottery tiebreaking. A leading example is the New York City public school district, which uses criteria like test scores and interviews to generate applicant rankings for some schools, combined with lottery tie-breaking at other schools. We develop methods that identify causal effects of assignment in such settings. Our approach generalizes the standard regression discontinuity design to allow for many running variables and treatments, some of which are randomly assigned. We show that lottery variation generates assignment risk at non-lottery programs for applicants away from nonlottery cutoffs, while non-lottery variation randomizes applicants near cutoffs regardless of lottery risk. These methods are applied to evaluate New York City's school progress assessments, which give schools letter grades as a summary measure of quality. Our estimates reveal that although Grade A schools boost achievement, these gains emerge only for students who attend lottery schools. Attendance at a coveted Grade A screened school, including some of the highest performing in the district, generates no measurable effects. Evaluation methods that fail to take advantage of both lottery and non-lottery variation miss this difference in impact.
BibTeX
@misc{abdulkadiroglu2017impact,
  author = {Atila Abdulkadiroğlu and Joshua D. Angrist and Yusuke Narita and Parag A. Pathak},
  title = {{Impact Evaluation in Matching Markets with General Tie-Breaking}},
  year = {2017},
  doi = {10.3386/w24172},
}
Figure