Model Mis-specification and Algorithmic Bias

Runshan Fu, Yangfan Liang, Peter ZhangarXiv 2021 · DOI · arXiv

Abstract

Machine learning algorithms are increasingly used to inform critical decisions. There is a growing concern about bias, that algorithms may produce uneven outcomes for individuals in different demographic groups. In this work, we measure bias as the difference between mean prediction errors across groups. We show that even with unbiased input data, when a model is mis-specified: (1) population-level mean prediction error can still be negligible, but group-level mean prediction errors can be large; (2) such errors are not equal across groups; and (3) the difference between errors, i.e., bias, can take the worst-case realization. That is, when there are two groups of the same size, mean prediction errors for these two groups have the same magnitude but opposite signs. In closed form, we show such errors and bias are functions of the first and second moments of the joint distribution of features (for linear and probit regressions). We also conduct numerical experiments to show similar results in more general settings. Our work provides a first step for decoupling the impact of different causes of bias.

Introduction

The prevailing narrative in AI ethics often centers on the "garbage in, garbage out" principle: biased outcomes are the result of skewed training data, sampling gaps, or historical prejudice embedded in labels. However, this research investigates a more fundamental source of unfairness: the mathematical structure of the machine learning process itself. Specifically, the authors explore model mis-specification—what happens when the chosen model (e.g., a linear regression) is insufficient to capture the true underlying data-generating process (DGP), even when the training data is perfectly representative and free of prejudice.

Methodology

The researchers isolated the effects of mis-specification using a combination of formal mathematical proofs and numerical simulations. They defined bias ($\tau$) as the difference in mean prediction errors between two demographic groups.

To ensure the results were not influenced by "dirty" data, the study employed an "open-box" approach using synthetic datasets where the ground truth was known. They simulated 10,000 observations for two equal-sized groups, generating feature vectors ($X_1, X_2$) from multivariate normal distributions and outcomes ($Y$) using predefined linear, logistic, and polynomial functions. This allowed them to compare correctly specified models (matching the true DGP) against mis-specified models (missing a variable or using the wrong functional form).

Key Results

The study reveals that model mis-specification creates systematic prediction disparities that are often hidden by population-level metrics.

  • The "Zero-Sum" Trap: When a model is mis-specified and groups are of equal size, the overall population-level error may appear negligible. However, this often masks a "worst-case" scenario where the mean error for one group is the exact opposite of the other. The model effectively balances its inaccuracies by over-predicting for one group and under-predicting for the other.
  • Redundant Encoding vs. Mis-specification: Contrary to common belief, features correlated with protected attributes (redundant encoding) do not inherently cause bias if the model is correctly specified. Bias only emerges when the model fails to capture the full relationship between features.
  • Empirical Evidence: As shown in Table 1, when a linear model is correctly specified with all relevant features ($X_1, X_2$), the bias ($\tau$) is nearly zero. However, omitting a single variable ($X_2$) causes the bias to skyrocket (e.g., from $2.28 \times 10^{-3}$ to $2.02$), even while population-level error remains low. Similarly, using a linear model for a polynomial DGP results in significant bias ($\tau = -0.752$).
Table 1: Numerical results showing the impact of model mis-specification on algorithmic bias (τ) across different data-generating processes and model types.

Discussion and Limitations

This work demonstrates that improving data quality is necessary but insufficient for ensuring algorithmic fairness. If a model is too simple for the reality it attempts to predict, it will likely be biased by design.

The authors acknowledge several limitations:

  1. Model Scope: Theoretical proofs are currently limited to linear and probit models, though numerical results suggest broader applicability to complex models like Random Forests.
  2. Risk Scores vs. Decisions: The focus is on raw output scores rather than final binary decisions, which involve thresholding that can introduce additional bias.
  3. Real-world Complexity: In practice, data bias and model mis-specification often occur simultaneously. Future research should investigate how these forces interact and amplify one another.

Ultimately, practitioners must recognize that model selection and feature engineering are ethical choices as much as technical ones.

Summary updated

Abstract

Machine learning algorithms are increasingly used to inform critical decisions. There is a growing concern about bias, that algorithms may produce uneven outcomes for individuals in different demographic groups. In this work, we measure bias as the difference between mean prediction errors across groups. We show that even with unbiased input data, when a model is mis-specified: (1) population-level mean prediction error can still be negligible, but group-level mean prediction errors can be large; (2) such errors are not equal across groups; and (3) the difference between errors, i.e., bias, can take the worst-case realization. That is, when there are two groups of the same size, mean prediction errors for these two groups have the same magnitude but opposite signs. In closed form, we show such errors and bias are functions of the first and second moments of the joint distribution of features (for linear and probit regressions). We also conduct numerical experiments to show similar results in more general settings. Our work provides a first step for decoupling the impact of different causes of bias.
BibTeX
@misc{fu2021model,
  author = {Runshan Fu and Yangfan Liang and Peter Zhang},
  title = {{Model Mis-specification and Algorithmic Bias}},
  year = {2021},
  eprint = {2105.15182},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.15182},
}
Figure