TL;DR
This paper introduces Bayesian hierarchical measurement models to combat "label bias"—the systematic error that occurs when we use proxy outcomes (like arrests or diagnoses) instead of the true outcomes we actually care about (like actual crime or underlying disease). By explicitly modeling the gap between the proxy and the truth, the authors show we can significantly improve prediction accuracy and reduce unfair disparities against specific demographic groups.
What Problem Does This Paper Address?
In many fields, the "ground truth" is invisible. A doctor doesn't see "diabetes status"; they see a "diagnosis." A sociologist doesn't see "criminal behavior"; they see "arrest records." These proxies are imperfect.
The danger arises when the error in the proxy is correlated with other variables. For example, if people without health insurance are less likely to be diagnosed with diabetes despite having the disease, a standard model trained on diagnosis data will systematically underpredict risk for the uninsured. This isn't just a math error; it’s a fairness issue that leads to underserved communities being further marginalized by "objective" algorithms. The authors tackle the challenge of how to build reliable models when our training labels are fundamentally "biased" by the measurement process itself.
Methods
The authors propose moving away from standard regression (which implicitly assumes the proxy is the truth) toward Bayesian Hierarchical Measurement Models.
The core idea is to treat the true outcome ($u$) as a latent (unobserved) variable. The model is structured into two parts:
- The Structural Model: How covariates (like age or neighborhood) relate to the latent true outcome.
- The Measurement Model: How the latent true outcome relates to the observed proxy label.
They highlight two specific architectures:
- Leakage Model (for Linear Regression): Used when the proxy "leaks" information about the latent state, often used for continuous or count data like crime rates.
- Threshold Model (for Logistic Regression): Used for binary outcomes. It assumes there are underlying continuous characteristics that, when they cross a certain threshold (which can vary by group, like insurance status), trigger an observed label (like a diagnosis).
Because these models are often "partially identified" (the data alone can't distinguish between the true effect and the measurement error), the authors emphasize using informative priors based on domain expertise and performing sensitivity analyses to see how results change under different assumptions about the measurement process.
Datasets
The paper evaluates the approach using two primary contexts:
- Simulated Criminal Justice Data: Based on a stylized model from previous literature (Zanger-Tishler et al., 2024), this dataset simulates individual criminal behavior and arrests across neighborhoods with varying levels of police enforcement. This allows the authors to compare their model against an "Oracle" that knows the simulated truth.
- NHANES (National Health and Nutrition Examination Survey): A real-world dataset containing both self-reported diabetes diagnoses (the proxy) and actual laboratory blood sugar measurements (the ground truth). This provides an empirically realistic test case for label bias in healthcare.
Key Insights and Ideas
- Explicit is Better than Implicit: Standard models have an implicit assumption that measurement error is random noise. Bayesian measurement models force researchers to make their assumptions about bias explicit and transparent.
- The Predictor Paradox: In the presence of label bias, adding more features to a standard model can actually make it less accurate and more unfair. The Bayesian approach mitigates this by accounting for why those features might be correlating with the error.
- Sensitivity as a Feature, Not a Bug: When domain knowledge is thin, these models allow practitioners to map out exactly how much their predictions depend on unknown measurement parameters. This helps gauge if a proxy is even "good enough" to use for high-stakes decisions.
- Fairness through Calibration: The authors demonstrate that "algorithmic fairness" is often a data-modeling problem. By adjusting for the fact that some groups are "measured" differently than others, the model naturally becomes more equitable.
Key Results
- Superior Accuracy: In the crime simulation, the Bayesian "Error Model" achieved a Root Mean Squared Error (RMSE) nearly identical to the "Oracle" model, significantly outperforming both simple and complex standard regressions.
- Eliminating Systematic Bias: In the same simulation, standard models showed a high correlation between prediction error and neighborhood policing. The Bayesian model's errors were uncorrelated with the neighborhood, meaning it didn't systematically over- or under-predict based on where someone lived.
- Healthcare Calibration: On the NHANES diabetes data, the measurement model was much better calibrated than standard logistic regression. It correctly identified risk levels for uninsured individuals, whereas the standard model systematically underpredicted their risk (matching the "Oracle" performance with a Log Score of -0.324 vs -0.333 for standard models).
Limitations and Future Work
- The Need for Domain Knowledge: These models aren't "plug-and-play." They require substantive knowledge about the measurement process (e.g., "how much less likely is an uninsured person to get a test?") to set effective priors.
- Identifiability: Some parameters simply cannot be learned from the data alone. The authors suggest "transparent parameterization" to separate what can be known from what must be assumed.
- Legal and Ethical Tensions: The authors acknowledge a "fundamental tension": to fix bias against a protected group, you often have to include that group's status in the model. This can run into legal doctrines regarding "disparate treatment," even if the goal is to ensure "equitable outcomes." Future work is needed to navigate these societal and legal hurdles.
Summary updated