When a machine learning model outputs biased predictions, the standard engineering response is to treat it as an optimization problem. We might inject a regularization term into the loss function, adjust classification thresholds, or re-weight the training samples. The implicit assumption behind these techniques is that bias is a software bug—a mathematical wrinkle that can be smoothed out with the right algorithmic constraint.
However, a cohesive body of research demonstrates that this model-centric view faces a fundamental ceiling. When unfairness is baked into the data-generating process—the historical, institutional, and environmental systems that produce the data—algorithmic interventions cannot achieve true equity. Instead of resolving the root causes of discrimination, purely mathematical adjustments merely obscure or redistribute the underlying disparities.
The Data-Generating Process as a Hard Ceiling
To understand why algorithmic fixes fail, we must look at where data originates. Datasets are not neutral reflections of reality; they are the footprints of historical social processes. As Jason Radford argues in Theory In, Theory Out: The uses of social theory in machine learning for social science, technical adjustments cannot rectify systemic biases that are entirely external to a model's mathematical architecture. When data reflects deeply entrenched inequalities, such as biased policing or discriminatory judicial practices, the resulting models will perpetuate these issues regardless of internal constraints.
In fact, this environmental constraint is so severe that even a completely random predictor can produce unfair outcomes if the underlying world lacks equal opportunity. In A New Paradigm for Counterfactual Reasoning in Fairness and Recourse, Lucius E.J. Bynum et al. demonstrate that focusing exclusively on model-level fairness ignores how the data-generating environment limits the opportunities available to certain groups. When the system itself is structurally skewed, optimizing the predictor alone is a superficial exercise. As Joshua R. Loftus et al. note in The long road to fairer algorithms, computational optimization often merely obscures structural inequities that actually require institutional or institutional reform.
Why Scale and Architecture Cannot Save Us
A common intuition in deep learning is that scaling up—gathering more data or building larger architectures—can overcome noise and bias. While this often holds true for performance accuracy, it fails for algorithmic fairness.
In tabular data, increasing the volume of training data does not mitigate algorithmic bias because disparities are frequently rooted in the base rates of social outcomes. As Frances Ding et al. show in Retiring Adult: New Datasets for Fair Machine Learning, structural inequalities scale alongside the dataset. Unlike domains where larger sample sizes minimize random error, tabular data fairness is constrained by persistent inequalities that simply become more precisely mapped as data grows.
Similarly, changing the model architecture does not resolve the issue. In a comprehensive evaluation of medical imaging foundation models, Ruinan Jin et al. discovered that demographic performance disparities remained remarkably consistent across 20 different popular foundation models. Their work, FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Models, suggests that biases are properties of the underlying data distributions rather than the models themselves. This is echoed by Hongbo Bo et al. in Failing on Bias Mitigation: A Case Study on the Challenges of Fairness in Government Data, who found that failures in bias mitigation are primarily driven by inherent data properties like historical biases and distribution shifts, rather than flaws in model architecture or fairness metrics.
The Mathematical Floor of Algorithmic Interventions
Why can't we simply regularize a model during training to ignore these base rate disparities? Corinna Hertweck et al. provide an information-theoretic explanation in Gradual (In)Compatibility of Fairness Criteria. They demonstrate that certain components of algorithmic unfairness are intrinsic to the dataset's historical collection process. These fixed biases represent a mathematical "floor" for unfairness; because they are properties of the data distribution rather than the model parameters, they cannot be eliminated through model-centric regularization techniques or fairness-aware constraints.
If we cannot fix the model during training, a popular alternative is post-processing—adjusting the final outputs, such as re-ranking recommendations or calibrating risk scores. However, the literature shows that post-processing is equally constrained by upstream data quality:
- Upstream Distortions Remain: Nicolas Usunier et al. demonstrate in Optimizing Generalized Gini Indices for Fairness in Rankings that applying fairness optimizations exclusively at the ranking or inference stage fails to mitigate biases established during initial data collection or preference estimation.
- Persistence of Pre-existing Unfairness: In Sampling Ex-Post Group-Fair Rankings, Sruthi Gorantla et al. note that post-processing techniques cannot rectify pre-existing unfairness embedded in the source data or labeling processes.
- Superficial Adjustments: Lujing Zhang et al. show in Fair Risk Control: A Generalized Framework for Calibrating Multi-group Fairness Risks that post-processing adjustments merely modify model outputs without addressing the fundamental biases originating from how a problem is defined or how data is gathered.
Shifting Focus to the Lifecycle and Environment
Because mathematical constraints on models and outputs are insufficient, meaningful mitigation requires shifting focus away from the model and toward the broader data lifecycle and structural environment.
First, we must treat data preparation as an active site of intervention. Jiongli Zhu et al. argue in Overcoming Data Biases: Towards Enhanced Accuracy and Reliability in Machine Learning that standard fairness interventions treat the "symptoms" of bias rather than the root causes. They advocate for using causal modeling during data cleaning and preparation to explicitly identify and rectify selection bias and confounding before modeling even begins.
Second, we must audit the structural environment being modeled. George C. Banks et al. point out in How to Reduce Bias in the Life Cycle of a Data Science Project that if a job's requirements or an environment's success criteria are inherently biased toward a specific demographic, purely mathematical adjustments to a selection model will fail.
This environment-first perspective is vividly illustrated by Jessy Xinyi Han et al. in Fairness Is More Than Algorithms: Racial Disparities in Time-to-Recidivism. By shifting the focus from binary outcomes to temporal dynamics in recidivism, the authors show that disparities emerging long after a model's prediction are driven by post-release systemic factors like housing and employment access. A mathematically fair algorithm cannot produce equitable social outcomes if it functions within a systemically biased infrastructure.
Traditional Model-Centric View:
[ Biased Data ] ──> [ Model + Algorithmic Constraint ] ──> [ "Fair" Output ]
│
(Fails due to data floor)
Structural Lifecycle View:
[ Social Environment Audit ] ──> [ Causal Data Cleaning ] ──> [ Model Training ] ──> [ Context-Aware Audit ]
Ultimately, addressing these root causes requires expanding who builds these systems and how they are evaluated. Caitlin Kuhlman et al. argue in No Computation without Representation: Avoiding Data and Algorithm Biases through Diversity that mathematical metrics fail because they treat datasets as neutral; avoiding these blind spots requires workforce diversity to challenge how data is defined and collected. To operationalize this, toolkits like the one proposed by Joachim Baumann et al. in FairnessLab: A Consequence-Sensitive Bias Audit and Mitigation Toolkit integrate principles from distributive justice to move beyond "one-size-fits-all" mathematical metrics, acknowledging that technical adjustments can never substitute for structural reforms.
Key Foundational Work
The long road to fairer algorithms (2020) — contributed 2 concept elements to this direction.
Theory In, Theory Out: The uses of social theory in machine learning for social science (2020) — contributed 2 concept elements to this direction.
Retiring Adult: New Datasets for Fair Machine Learning (2021) — contributed 2 concept elements to this direction.