A calibrated BISG for inferring race from surname and geolocation

Philip Greengard, Andrew GelmanJournal of the Royal Statistical Society Series A (Statistics in Society) 2023 · arXiv

Abstract

Bayesian Improved Surname Geocoding (BISG) is a ubiquitous tool for predicting race and ethnicity using an individual's geolocation and surname. Here we demonstrate that statistical dependence of surname and geolocation within racial/ethnic categories in the United States results in biases for minority subpopulations, and we introduce a raking-based improvement. Our method augments the data used by BISG—distributions of race by geolocation and race by surname—with the distribution of surname by geolocation obtained from state voter files. We validate our algorithm on state voter registration lists that contain self-identified race/ethnicity.

The Problem: The Independence Assumption

In critical fields like lending, healthcare, and election law, measuring racial disparities is essential but often hindered by a lack of direct data. Researchers frequently rely on Bayesian Improved Surname Geocoding (BISG) to estimate race probabilities. However, standard BISG relies on a "conditional independence" assumption: it assumes that given a person's race, their surname and geolocation are independent factors.

In reality, this assumption is false. Ethnic surnames often cluster in specific neighborhoods. By ignoring these local clusters, standard BISG systematically undercounts minority groups in many areas, leading to biased results in the very fairness audits it is meant to support. For example, in Florida, standard BISG was found to underestimate Hispanic populations in 88% of counties.

Methodology: The Raking Innovation

To address these systematic biases, we propose a calibrated version of BISG using a statistical technique called raking (or iterative proportional fitting). While standard BISG uses two "margins" from the Census (race by surname and race by geolocation), our innovation augments this with two additional data sources:

  1. Joint Surname-Geolocation Distribution: We use state voter registration files to determine the exact number of people with a specific surname living in a specific county.
  2. Statewide Race Totals: We use the Current Population Survey (CPS) to establish an accurate "target" for the total racial breakdown of a state.

The algorithm starts with standard BISG predictions and mathematically adjusts (rakes) them until the results match these known margins. This ensures that final predictions are consistent with both local neighborhood data and overall state demographics.

Datasets and Validation

The study validates this method using 2020 voter registration lists from Florida and North Carolina. These datasets are unique because they include self-identified race/ethnicity, providing a "ground truth" for evaluation. These were combined with U.S. Census data and the CPS Voter Supplement to provide the necessary calibration margins.

Key Results

The improvement in accuracy is substantial across all tested states. The raking method is "unbiased by design," meaning the mean error across an entire state is zero.

  • Florida (2020): The mean absolute error for the raking method was only 4%, compared to 10% for standard BISG.
  • North Carolina (2020): Raking achieved a mean absolute error of 3%, while standard BISG lagged at 7%.
Error and relative error comparison for Florida (2020) across racial groups. Standard BISG (red) shows significant bias and higher error compared to the raking method (blue).

As shown in the figures, standard BISG consistently struggles with "relative accuracy" for smaller subpopulations, such as Asian and Pacific Islanders (API), often underestimating them by 25% or more. The raking method acts as a "safety rail," preventing the model from drifting into biased territory.

Error and relative error comparison for North Carolina (2020). The raking method (blue) maintains lower error and better calibration across all racial categories compared to standard BISG (red).

Limitations and Future Work

While highly effective, the method relies on the availability of joint surname-geolocation data. While accessible via voter files in many states, this data may be harder to obtain in other administrative domains. Future work could improve accuracy further by using finer-grained geographic data (such as census tracts) and incorporating additional features like first names or sex into the predictive model.

Summary updated

Abstract

Abstract Bayesian Improved Surname Geocoding (BISG) is a ubiquitous tool for predicting race and ethnicity using an individual’s geolocation and surname. Here we demonstrate that statistical dependence of surname and geolocation within racial/ethnic categories in the US results in biases for minority subpopulations, and we introduce a raking-based improvement. Our method augments the data used by BISG—distributions of race by geolocation and race by surname—with the distribution of surname by geolocation obtained from state voter files. We validate our algorithm on state voter registration lists that contain self-identified race/ethnicity.
BibTeX
@article{greengard2023calibrated,
  author = {Philip Greengard and Andrew Gelman},
  title = {{A calibrated BISG for inferring race from surname and geolocation}},
  journal = {Journal of the Royal Statistical Society Series A (Statistics in Society)},
  year = {2023},
}
Figure