Imagine an engineer tasked with making a predictive model "fair." The standard engineering playbook is straightforward: select a mathematical metric—such as equalizing false positive rates across demographic groups—and optimize the model's decision thresholds until that constraint is satisfied. On paper, the math balances perfectly.
Yet, when deployed in the real world, these technically optimized models frequently fail to achieve actual justice. They can inadvertently reinforce systemic discrimination, disrupt public safety, or alienate the very communities they were designed to protect.
This disconnect is not a software bug that can be patched with better code. It represents a fundamental sociotechnical fairness gap: a mismatch between the rigid, quantifiable logic of algorithms and the fluid, historical, and power-laden reality of human social systems.
The Formalism Trap and Mathematical Reductionism
At the heart of this gap lies what Andrus et al. term the formalism trap—the mistaken belief that complex human values like fairness, equity, or safety can be entirely encapsulated by mathematical formulas.
Algorithms inherently require explicit, quantifiable constraints to operate. However, as Barocas et al. argue, the process of translating broad social objectives into rigid mathematical constraints strips away essential historical and situational context. When we reduce fairness to a statistical parity metric, we treat data as a neutral representation of the world, ignoring the messy social processes and structural inequalities that generated that data in the first place (Jagadish et al.).
[Complex Social Reality] ──(Reductionism)──> [Rigid Mathematical Proxy]
│ │
(Historical context, (Optimizes numbers,
power dynamics, fluid values) ignores systemic roots)
This reductionism creates a false sense of equity. For example, Corbett-Davies et al. mathematically demonstrated that satisfying common definitions of algorithmic fairness requires applying different risk thresholds to different groups, which can paradoxically reduce overall public utility. Because mathematical optimization operates strictly within the boundaries of the data provided, it cannot address or mitigate the underlying social drivers of inequality.
Why Optimization Fails to Address Power Dynamics
When algorithmic fairness is treated as an isolated optimization problem, it often treats bias as a superficial "bug" rather than a manifestation of deeper sociological hierarchies.
By "studying up" on industry practices, Papakyriakopoulos et al. illustrated how commercial algorithms actively reproduce historical patriarchal and white supremacist structures by generating status-oriented labels for dominant groups while stereotyping marginalized groups. Because these biases are baked into the cultural and structural fabric of the training data, optimizing for statistical parity across groups does nothing to dismantle the underlying power asymmetries (McFadden et al.).
Instead, technical de-biasing can act as a performative distraction. It allows organizations to claim compliance with abstract ethical principles while leaving harmful institutional practices and business models completely unchallenged (Barocas et al.).
The Rigidity of Code vs. The Fluidity of Justice
Why can't computer scientists simply engineer a more sophisticated, universal formula for fairness? The answer lies in the fundamentally different ways that algorithms and human institutions process justice.
| Algorithmic Fairness | Sociotechnical Fairness |
|---|---|
| Rigid & Static: Hard-coded into optimization objectives and error-rate constraints. | Contextual & Fluid: Dependent on evolving social norms, historical context, and qualitative nuances. |
| Decontextualized: Treats demographic categories as discrete, independent variables. | Relational: Embedded within institutional power dynamics and lived human experiences. |
| Mathematical Consensus: Requires a single, mathematically consistent objective function. | Pluralistic: Characterized by competing, often irreconcilable stakeholder values. |
Consider how law handles equity. As Wachter et al. observe, European Union non-discrimination law deliberately avoids rigid mathematical definitions in favor of "contextual equality." This principle grants legal authorities the flexibility to interpret justice dynamically based on the specific facts of a case and shifting societal values. Hard-coding a specific fairness metric into an automated system strips away this interpretive flexibility, effectively transferring normative and political decision-making power from democratic judiciaries to software engineers.
Furthermore, human notions of fairness are deeply subjective and resistant to mathematical consensus. When Cheng et al. solicited fairness perspectives from real-world stakeholders in child welfare systems, they found that practitioners' and families' ethical values were deeply context-dependent and could not be reconciled into a single statistical metric like equalized odds. Similarly, Nurock emphasizes that purely technical metrics lack the capacity to incorporate the relational ethics and "ethics of care" required to maintain pluralistic public discourse and healthy human relationships.
Moving Beyond Techno-Solutionism
To escape the formalism trap, researchers argue for a fundamental shift away from techno-solutionism—the reductionist belief that complex structural problems can be solved unilaterally with an algorithmic fix (Easterbrook).
As Dolata et al. point out, algorithmic fairness research has historically been siloed: technical solutions lack real-world social context, while social critiques lack technical execution. Bridging the sociotechnical gap requires evaluating AI systems not as isolated models, but as holistic systems embedded within human organizational structures.
Practically, this means moving beyond code-based debugging toward frameworks that integrate disaggregated evaluations, organizational governance, and deep community trust (Weerts et al.). True fairness cannot be automated; it requires evaluating the broader environment in which an AI system operates, ensuring that technical tools serve democratic accountability rather than replacing it.
Key Foundational Work
Algorithmic decision making and the cost of fairness (2017) — contributed 1 concept elements to this direction.