Hierarchical Bayesian Models to Mitigate Systematic Disparities in Prediction with Proxy Outcomes

Jonas M. Mikhaeil, Andrew Gelman, Philip Greengard

Abstract

Label bias occurs when the outcome of interest is not directly observable and instead, modeling is performed with proxy labels. When the difference between the true outcome and the proxy label is correlated with predictors, this can yield systematic disparities in predictions for different groups of interest. We propose Bayesian hierarchical measurement models to address these issues. When strong prior information about the measurement process is available, our approach improves accuracy and helps with algorithmic fairness. If prior knowledge is limited, our approach allows assessment of the sensitivity of predictions to the unknown specifications of the measurement process. This can help practitioners gauge if enough substantive information is available to guarantee the desired accuracy and avoid disparate predictions when using proxy outcomes. We demonstrate our approach through practical examples.

Introduction: The Challenge of Label Bias

In many critical domains, the "ground truth" outcome we wish to predict is fundamentally unobservable. Instead, researchers must rely on proxy outcomes: a doctor observes a medical diagnosis rather than the underlying disease state; a sociologist observes an arrest record rather than the actual criminal behavior.

The danger of using these proxies arises when the measurement error is not random but correlated with other variables. For instance, if individuals without health insurance are less likely to receive a formal diagnosis despite having a disease, a standard model trained on diagnosis data will systematically underpredict risk for the uninsured. This "label bias" creates a significant fairness issue, where underserved communities are further marginalized by seemingly objective algorithms. This paper addresses how to build reliable, equitable models when training labels are fundamentally biased by the measurement process.

Methods: Bayesian Hierarchical Measurement Models

To combat label bias, we propose a shift from standard regression—which assumes proxies are ground truth—to Bayesian Hierarchical Measurement Models. This approach treats the true outcome ($u$) as a latent, unobserved variable and structures the model into two distinct components:

  1. The Structural Model: Defines how covariates (e.g., age, neighborhood) relate to the latent true outcome.
  2. The Measurement Model: Defines the relationship between the latent true outcome and the observed proxy label.
Architectures for the proposed Measurement, Leakage, and Threshold models.

We highlight two specific architectures for different data types. The Leakage Model is designed for continuous or count data (like crime rates) where the proxy "leaks" information about the latent state. The Threshold Model is intended for binary outcomes (like medical diagnoses), assuming underlying continuous characteristics trigger an observed label only after crossing a group-specific threshold.

Because these models are often only partially identified by the data alone, we emphasize the use of informative priors based on domain expertise and sensitivity analyses to map how predictions shift under varying assumptions about the measurement process.

Experimental Evaluation

We evaluated our approach using two primary contexts:

  • Simulated Criminal Justice Data: A stylized model simulating individual criminal behavior and arrests across neighborhoods with varying police enforcement. This allowed for a direct comparison against an "Oracle" model that knows the simulated truth.
  • NHANES (National Health and Nutrition Examination Survey): A real-world dataset containing both self-reported diabetes diagnoses (the proxy) and laboratory blood sugar measurements (the ground truth), providing an empirically realistic test case for healthcare label bias.

Key Results

Our Bayesian "Error Model" demonstrated superior performance across multiple metrics. In the criminal justice simulation, the model achieved a Root Mean Squared Error (RMSE) nearly identical to the Oracle model, significantly outperforming both simple and complex standard regressions as the bias parameter ($\beta$) increased.

RMSE comparison showing the Bayesian Error Model's performance relative to Oracle and standard models across varying levels of bias (beta).

Furthermore, the Bayesian approach effectively eliminated systematic bias. While standard models showed a high correlation between prediction error and neighborhood policing levels, the Bayesian model's errors were uncorrelated with the neighborhood. In the NHANES healthcare application, the measurement model was significantly better calibrated than standard logistic regression, correctly identifying risk levels for uninsured individuals where standard models systematically underpredicted risk (matching Oracle performance with a Log Score of -0.324 vs -0.333).

Discussion and Limitations

By making assumptions about measurement error explicit rather than implicit, Bayesian measurement models mitigate the "Predictor Paradox," where adding more features to a standard model can actually decrease accuracy and increase unfairness.

However, these models are not "plug-and-play." They require substantive domain knowledge to set effective priors. Additionally, there is a fundamental tension between technical and legal frameworks: to correct bias against a protected group, that group's status must often be included in the model, which can conflict with legal doctrines regarding disparate treatment. Future work must navigate these societal and legal hurdles to ensure that technical solutions for algorithmic fairness can be effectively implemented.

Summary updated

Abstract

Label bias occurs when the outcome of interest is not directly observable and instead, modeling is performed with proxy labels. When the difference between the true outcome and the proxy label is correlated with predictors, this can yield systematic disparities in predictions for different groups of interest. We propose Bayesian hierarchical measurement models to address these issues. When strong prior information about the measurement process is available, our approach improves accuracy and helps with algorithmic fairness. If prior knowledge is limited, our approach allows assessment of the sensitivity of predictions to the unknown specifications of the measurement process. This can help practitioners gauge if enough substantive information is available to guarantee the desired accuracy and avoid disparate predictions when using proxy outcomes. We demonstrate our approach through practical examples.
BibTeX
@article{mikhaeil2024hierarchical,
  author = {Jonas M. Mikhaeil and Andrew Gelman and Philip Greengard},
  title = {{Hierarchical Bayesian Models to Mitigate Systematic Disparities in Prediction with Proxy Outcomes}},
  journal = {Journal of the Royal Statistical Society Series A (Statistics in Society)},
  year = {2024},
  doi = {10.1093/jrsssa/qnae142},
}
Figure