Questioning the assumptions behind fairness solutions

Rebekah Overdorf, Bogdan Kulynych, Ero Balsa, Carmela Troncoso, Seda F. GürsesarXiv 2018 · OpenReview

Introduction

The prevailing narrative in the Fairness, Accountability, and Transparency (FAccT) movement suggests that algorithmic bias is a technical bug—a flaw in design that can be patched from within by the companies that build them. This "solution by design" approach assumes that if service providers are given the right tools and metrics, they will naturally move toward more equitable outcomes. However, this paper challenges that fundamental assumption, arguing that many algorithmic harms are not accidental bugs but are instead baked into the profit-driven incentives of modern Optimization System Providers (OSPs).

The Structural Limits of Internal Fairness

Modern OSPs, such as Uber, Waze, or Facebook, are not static information systems; they are dynamic entities that constantly sense their environment and optimize for specific objective functions. This operational model creates two significant barriers to internal fairness solutions:

1. Misaligned Incentives

For many OSPs, "unfairness" is often a byproduct of successful optimization. If routing traffic through a quiet residential neighborhood or suppressing driver wages increases the bottom line, the provider has little financial incentive to "fix" the algorithm. In these cases, the harm is a feature of the system's efficiency, not a failure of its design.

2. Environmental Externalities

Optimization systems frequently impact individuals who are not "users" of the platform. For example, Waze may flood a street with traffic, harming residents who have never downloaded the app. Because these individuals do not interact with the system, the OSP lacks the sensors to capture data on the harm being caused. Consequently, internal fairness metrics—which rely on user data—are mathematically incapable of addressing these externalized costs.

The "Accident" Fallacy and Fairness as Distraction

The authors argue that treating algorithmic bias as a mere "accident" is a fallacy. Many harms are inherent to the objective function, such as maximizing ad revenue or minimizing operational costs. Furthermore, focusing solely on "fairness" can sometimes distract from deeper systemic issues. For instance, ensuring that suppressed wages are distributed "fairly" among gig workers ignores the fact that the system is optimized to keep those wages as low as possible for everyone. In this context, distributing a "fair share" of an exploitative outcome is not a meaningful victory for the subjects of the algorithm.

Protective Optimization Technologies (POTs)

Given that we cannot always trust OSPs to be honest or effective in self-regulation, the authors propose a shift in focus: from internal fixes to external defenses. They introduce the concept of Protective Optimization Technologies (POTs)—tools that allow communities and users to intervene in optimization systems from the outside.

The paper categorizes these external interventions into several primary strategies:

  • Data Poisoning: Subjects can manipulate the inputs a system receives to force a different, more desirable outcome. An example includes residents reporting fake road closures to navigation apps to redirect traffic away from their neighborhoods.
  • Coordinated Action: Users can act in concert to bypass or manipulate system constraints. For example, Uber drivers have been known to simultaneously turn off their apps to trigger surge pricing, effectively "hacking" the supply-and-demand algorithm to ensure a livable wage.
  • Adversarial Perturbations: Utilizing techniques from adversarial machine learning, users can "pollute" their digital profiles. Browser plugins that click on random ads to confuse trackers render the underlying optimization (targeted advertising) useless, protecting user privacy through noise.

Real-World Case Studies

The paper illustrates these concepts through several qualitative examples:

  • Waze: Demonstrates how "optimal" routing for users creates negative externalities like noise and congestion for non-users, highlighting the system's blindness to the physical environment.
  • Pokémon Go: Shows how data imbalances (e.g., fewer game assets spawning in minority neighborhoods) reflect and reinforce existing social inequalities, which traditional data pre-processing struggles to resolve.
  • Facebook: Points to instances of inflated video viewing metrics to demonstrate that even powerful stakeholders cannot always rely on OSPs to provide honest data when financial incentives are high.

Conclusion and Future Directions

The shift toward POTs represents a move from "guerrilla" tactics to a formalized field of defensive technology. While current POTs are often ad-hoc, the authors call for a research agenda that focuses on:

  • Formalizing POTs: Moving from "hacky" interventions to mathematically grounded defensive tools that provide predictable protections.
  • Incentive Alignment: Exploring how democratic governance, taxation, and regulation can force OSPs to internalize the costs of their externalities.
  • Holistic Modeling: Moving away from static decision-making models toward those that account for the long-term interaction between algorithms and the environments they inhabit.

Ultimately, the paper argues that fairness is not just a technical property to be tuned by developers, but a site of contestation that requires empowering the public to defend themselves against the systems that optimize their lives.

Summary updated

Abstract

In addition to their benefits, optimization systems can have negative economic, moral, social, and political effects on populations as well as their environments. Frameworks like fairness have been proposed to aid service providers in addressing subsequent bias and discrimination during data collection and algorithm design. However, recent reports of neglect, unresponsiveness, and malevolence cast doubt on whether service providers can effectively implement fairness solutions. These reports invite us to revisit assumptions made about the service providers in fairness solutions. Namely, that service providers have (i) the incentives or (ii) the means to mitigate optimization externalities. Moreover, the environmental impact of these systems suggests that we need (iii) novel frameworks that consider systems other than algorithmic decision-making and recommender systems, and (iv) solutions that go beyond removing related algorithmic biases. Going forward, we propose Protective Optimization Technologies that enable optimization subjects to defend against negative consequences of optimization systems.
BibTeX
@misc{overdorf2018questioning,
  author = {Rebekah Overdorf and Bogdan Kulynych and Ero Balsa and Carmela Troncoso and Seda F. Gürses},
  title = {{Questioning the assumptions behind fairness solutions}},
  year = {2018},
  eprint = {1811.11293},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.11293},
}
Figure