Accurate Fairness: Improving Individual Fairness without Trading Accuracy

Xuran Li, Peng Wu, Jing SuAAAI 2023 · DOI

TL;DR

This paper introduces "accurate fairness," a novel criterion that aligns individual fairness with predictive accuracy by ensuring similar individuals receive treatments consistent with the ground truth. They propose a Siamese neural network approach to optimize this criterion, demonstrating improved individual fairness and accuracy simultaneously, unlike previous methods that often involve a trade-off.

What Problem Does This Paper Address?

Machine learning models are increasingly used for critical decisions, but ensuring both accuracy and fairness is a significant challenge. Traditional fairness metrics often focus on group-level equality, which can mask discrimination against individuals within those groups. Conversely, focusing solely on individual fairness can lead to inaccurate predictions. This paper tackles the fundamental incompatibility between accuracy and individual fairness, aiming to achieve both without compromise. The authors highlight that existing methods often force a trade-off: improving one aspect may degrade the other, leading to either "false fairness" (fair predictions for unqualified individuals) or "true bias" (accurate but discriminatory predictions).

Methods

The core of the paper is the introduction of Accurate Fairness, a new criterion that refines individual fairness. It posits that for any individual and their "similar counterparts" (individuals identical except for sensitive attributes), their predictions should uniformly conform to the individual's ground truth. This is formalized by bounding the prediction error relative to the ground truth by a constant factor related to the distance between individuals.

To implement this, the authors propose a Siamese fairness in-processing approach. This method trains a machine learning model by simultaneously processing multiple similar individuals. The Siamese architecture allows the model to learn a shared representation and optimize for both accuracy and the accurate fairness constraint. This contrasts with traditional methods that often optimize accuracy and fairness separately or sequentially. They also introduce fairness confusion matrix-based metrics (fair-precision, fair-recall, fair-F1 score) to quantify the trade-off between accuracy and individual fairness.

Datasets

The paper uses the following datasets for evaluation:

  • Adult (Census Income): A popular dataset for fairness research.
  • German Credit: Another standard dataset for evaluating fairness in credit scoring.
  • ProPublica Recidivism (COMPAS): A well-known dataset for assessing fairness in criminal justice risk assessment.
  • Ctrip Dataset: A real-world dataset from Ctrip, a travel service provider, used to investigate service discrimination based on customer consumption habits.

Key Insights and Ideas

  • Accurate Fairness as a Win-Win: The central idea is that by focusing on treating similar individuals uniformly according to the ground truth, we can simultaneously improve both accuracy and individual fairness. This challenges the prevailing notion of an inherent trade-off.
  • Refining Individual Fairness: The paper argues that existing individual fairness definitions might be too broad. By requiring similarity within sub-populations defined by non-sensitive attributes, accurate fairness is more stringent and less prone to "fair but faulty" predictions.
  • Siamese Networks for Holistic Fairness: The application of Siamese networks is a novel approach to bias mitigation. By processing similar individuals in parallel, the network can learn to enforce the accurate fairness constraints more effectively than sequential processing.
  • Fairness Confusion Matrix: This new metric provides a more nuanced understanding of fairness performance beyond simple accuracy or group fairness metrics, explicitly highlighting the interplay between accuracy and individual fairness.

Key Results

  • Improved Individual Fairness and Accuracy: The Siamese fairness approach, on average, achieved 1.02%-8.78% higher individual fairness (fairness through awareness) and 8.38%-13.69% higher accuracy compared to state-of-the-art bias mitigation techniques across popular fairness datasets.
  • Higher True Fair Rate and F-F1 Score: The method resulted in 10.09%-20.57% higher true fair rates and 5.43%-10.01% higher fair-F1 scores, indicating a better balance between accuracy and individual fairness.
  • Mitigation of Service Discrimination: Applied to the Ctrip dataset, the approach significantly reduced service discrimination, serving 112.33% more customers fairly and, importantly, serving 81.29% more customers in an accurately fair way.
  • Group Fairness Implication: The paper theoretically shows that accurate fairness implies certain group fairness criteria (statistical parity and confusion matrix-based fairness) over unions of similar sub-populations.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, Siamese networks can be computationally intensive, especially with large sub-populations.
  • Definition of "Similar Individuals": The paper relies on the definition of similar individuals differing only on sensitive attributes. Future work could explore more nuanced definitions of similarity.
  • Sensitive Attribute Identification: The current approach assumes sensitive attributes are known. Future research could investigate how to identify and mitigate bias when sensitive attributes are not explicitly provided.
  • Diagnosing Bias Sources: The authors suggest adapting the fairness confusion matrix to analyze which sensitive attributes contribute most to bias and to diagnose which groups are treated unfavorably.
  • Generalization to Other Fairness Notions: Exploring how accurate fairness can refine other individual and group fairness criteria is a potential future direction.

Summary updated

BibTeX
@inproceedings{li2023accurate,
  author = {Xuran Li and Peng Wu and Jing Su},
  title = {{Accurate Fairness: Improving Individual Fairness without Trading Accuracy}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i12.26674},
}
Figure