Traditional machine learning fairness focuses on mathematical parity—ensuring that error rates or selection frequencies are equal across demographic groups. However, optimizing blindly for these statistical targets can backfire. If a model achieves fairness by simply reducing the accuracy or quality of decisions for a well-performing group without helping anyone else, it falls prey to what philosophers call the "leveling down objection."
To solve this, a growing body of research shifts the focus from abstract statistical properties to the tangible, real-world consequences of decisions. This approach is known as consequentialist or utility-based algorithmic fairness.
The Failure of Count-Based Metrics
Why do standard metrics fail? They treat all errors and outcomes as having equal weight. In reality, a false positive in a medical setting (e.g., an unnecessary treatment with mild side effects) has a completely different impact than a false negative (e.g., a missed life-threatening diagnosis).
As Heidari et al. point out, frameworks that ignore the distinction between beneficial and harmful prediction errors risk harming the very groups they intend to protect. By shifting from binary decision counts to expected utility functions, we can capture the actual well-being distributed to individuals. Baumann et al. argue that evaluating fairness based on decision utility rather than binary outcomes addresses the leveling down objection by ensuring that fairness is achieved through the distribution of positive impact rather than merely equalizing error rates or rejection frequencies.
Mapping Decisions to Utility Matrices
Mechanically, this shift requires replacing simple confusion matrices with outcome utility matrices. Instead of evaluating a model based on how often it is right or wrong, researchers assign specific values to the consequences of classification results.
Baumann et al. demonstrate that this approach allows fairness evaluations to reflect the actual benefits or harms experienced by individuals, such as weighing the physical harm of medical side effects against the benefit of treatment. Similarly, Pichler et al. introduce a framework where "happiness"—a measure of group-level utility—is optimized using linear programming, ensuring that value, rather than just labels, is equitably distributed.
To make these utilities accurate, they must be grounded in what stakeholders actually care about. Finocchiaro et al. advocate for preference-informed fairness, which replaces fixed classification targets with individual utility functions to account for the heterogeneous impact of decisions across different users.
Standard Fairness:
[Decision Output] ---> Equalize Error Rates Across Groups (Counts)
Consequentialist Fairness:
[Decision Output] ---> [Utility Matrix (Harms/Benefits)] ---> Equalize/Optimize Realized Value
Real-World Trade-Offs and Value-Weighted Auditing
In practical applications like finance or fraud detection, count-based metrics obscure massive disparities. For instance, in transaction fraud, blocking a $10,000 legitimate transaction causes far more harm to a user than blocking a $10 transaction.
Kamalaruban et al. propose value-weighted parity metrics that assess fairness based on the cumulative monetary value of protected assets rather than raw counts. To operationalize this in industry, McCalman et al. introduce the FEAT methodology, which inverts the traditional pipeline: instead of picking a mathematical metric a priori, system owners derive metrics by explicitly mapping out the distribution of potential harms and benefits across demographic groups.
Accounting for utility weights becomes even more critical when auditing for intersectional groups. Davidson demonstrates a startling "fairness paradox": algorithms optimized for fairness on individual attributes often significantly increase unfairness for intersectional subgroups when evaluated using count-based metrics. By assigning weights to different output categories to reflect their relative utility, audits can uncover hidden disparities in the actual value distributed to these subgroups.
Optimizing for Distributive Justice
By framing fairness around utility, machine learning can directly inherit well-studied frameworks from welfare economics and mechanism design. Coots et al. and Nilforoshan et al. both advocate for a consequentialist framework that models decision problems using utility functions and budget constraints to explicitly balance competing objectives and understand real-world trade-offs.
When optimizing these distributions, researchers look to economic inequality measures. Diao et al. reframe group fairness from a sparsity perspective, utilizing economic metrics like the Gini index to ensure that transferring performance to a lower-performing group strictly improves the fairness score. In spatial settings, Walsh shows that minimizing the Gini index of utilities (user preferences) rather than raw physical distances avoids counterintuitive penalties and aligns outcomes with actual user benefits.
Ultimately, as Weerts et al. argue, justifying fairness metrics requires looking beyond the historical causes of inequality to evaluate the actual downstream utility of outcomes, protecting systems from falling victim to Goodhart's Law where the metric ceases to be a good measure of fairness once it becomes a target.
Key Foundational Work
Bridging Machine Learning and Mechanism Design towards Algorithmic Fairness (2021) — contributed 1 concept elements to this direction.
A Moral Framework for Understanding Fair ML through Economic Models of Equality of Opportunity (2019) — contributed 1 concept elements to this direction.