Poisoning Attacks on Algorithmic Fairness

David Solans, Battista Biggio, Carlos Castillo

Abstract

As machine learning models are increasingly integrated into critical social decision-making processes—such as credit scoring, hiring, and recidivism prediction—ensuring their fairness has become a paramount security concern. While existing adversarial machine learning research primarily focuses on compromising model integrity (misclassification) or availability (denial of service), this paper introduces the concept of fairness poisoning. We demonstrate that a malicious actor can strategically inject a small amount of "poisoned" data into a training set to induce systematic bias against specific demographic groups. Crucially, these attacks are designed to be stealthy, maintaining high overall accuracy to avoid detection by standard monitoring tools. We present a gradient-based optimization framework for crafting these attacks and evaluate their impact on both synthetic and real-world datasets.

Introduction

The rapid adoption of algorithmic decision-making has sparked significant interest in algorithmic fairness. Most existing literature assumes that bias in models arises unintentionally from historical prejudices present in the training data. However, this paper identifies a critical security gap: the potential for intentional corruption of fairness.

An attacker may seek to manipulate a model to disadvantage a specific protected group (e.g., based on race or gender) for ideological or competitive reasons. Unlike traditional poisoning attacks that aim to degrade overall performance, a fairness poisoning attack seeks to maximize a disparity metric (like demographic parity or equalized odds) while keeping the model's error rate within acceptable bounds. This "stealthy" nature makes fairness poisoning a particularly insidious threat.

Methodology: Bilevel Optimization for Fairness Poisoning

The authors propose a novel framework for generating poisoned samples by formulating the attack as a bilevel optimization problem.

In this framework, the attacker's goal is to find a set of poisoned points $D_p$ that, when added to the clean training set $D_{tr}$, results in a model $\theta^*$ that maximizes an unfairness objective $\mathcal{L}_{atk}$ on a validation set $D_{val}$.

  1. Inner Loop: The model parameters $\theta^*$ are obtained by minimizing the training loss on the poisoned dataset: $\theta^* = \arg\min_\theta \mathcal{L}(D_{tr} \cup D_p, \theta)$.
  2. Outer Loop: The attacker optimizes the features of the poisoned points to maximize the unfairness: $\max_{D_p} \mathcal{L}_{atk}(D_{val}, \theta^*)$.

To solve this efficiently, the authors employ a gradient-based approach. By leveraging the Karush-Kuhn-Tucker (KKT) conditions of the inner optimization, they can compute the gradient of the optimal model parameters $\theta^*$ with respect to the poisoned inputs $D_p$. This allows the attacker to iteratively "nudge" the poisoned samples in the feature space to maximize the target disparity.

Heatmaps of the attacker's loss and disparate impact across the feature space, illustrating the optimization landscape for crafting poisoned samples.

The attack can be executed in a white-box setting, where the attacker has full knowledge of the training pipeline, or a black-box setting, where the attacker optimizes against a surrogate model and relies on the transferability of the poisoned samples to the target model.

Experimental Setup

The effectiveness of the attack was evaluated using two primary datasets:

  • Synthetic Data: A 2,000-sample dataset with two-dimensional features and a binary sensitive attribute. This controlled environment allowed for precise measurement of how initial group separation affects attack success.
  • COMPAS Dataset: A standard benchmark in fairness research containing criminal recidivism data. It includes 18 features for over 6,000 offenders, with race and gender serving as sensitive attributes.

The study tested several common classifiers, including Logistic Regression, Support Vector Machines (SVMs), Naive Bayes, Decision Trees, and Random Forests.

Results and Analysis

The experiments yielded several key insights into the vulnerability of machine learning models to fairness poisoning.

Impact on Decision Boundaries

On synthetic data, the gradient-based attack successfully shifted decision boundaries to increase group disparity. As shown in the comparison below, the injection of poisoned points (represented by stars) forces the model to adopt a boundary that disproportionately misclassifies members of the unprivileged group compared to the privileged group, even when the overall accuracy remains relatively stable.

Comparison of decision boundaries on synthetic data before (left) and after (right) a fairness poisoning attack. The poisoned points (stars) shift the boundary to increase group disparity.

Stealth and Transferability

A major finding is that fairness can be significantly compromised with only a moderate impact on accuracy. This confirms the attacker's ability to remain stealthy. Furthermore, the attacks demonstrated high transferability; poisoned samples generated for a Logistic Regression model were often effective at inducing bias in an SVM model. This suggests that attackers do not need perfect knowledge of the target system to be successful.

Model Resilience

While linear models like Logistic Regression and SVMs were highly susceptible, non-linear models like Decision Trees and Random Forests exhibited greater resilience. However, they were not entirely immune, and the authors suggest that more sophisticated attack strategies might still be effective against them.

Conclusion and Future Work

This research highlights a new and critical threat vector in adversarial machine learning. Fairness is not just a statistical property to be monitored; it is a security property that must be defended. The success of these poisoning attacks, even in black-box scenarios, underscores the need for robust training algorithms and data-sanitization filters.

Future work should explore more complex constraints on poisoned samples to evade outlier detection and investigate the impact of these attacks on deep neural networks. Additionally, developing defenses that can distinguish between natural data shifts and intentional fairness poisoning is a vital next step for the community.

Summary updated

Abstract

Research in adversarial machine learning has shown how the performance of machine learning models can be seriously compromised by injecting even a small fraction of poisoning points into the training data. While the effects on model accuracy of such poisoning attacks have been widely studied, their potential effects on other model performance metrics remain to be evaluated. In this work, we introduce an optimization framework for poisoning attacks against algorithmic fairness, and develop a gradient-based poisoning attack aimed at introducing classification disparities among different groups in the data. We empirically show that our attack is effective not only in the white-box setting, in which the attacker has full access to the target model, but also in a more challenging black-box scenario in which the attacks are optimized against a substitute model and then transferred to the target model. We believe that our findings pave the way towards the definition of an entirely novel set of adversarial attacks targeting algorithmic fairness in different scenarios, and that investigating such vulnerabilities will help design more robust algorithms and countermeasures in the future.
BibTeX
@inproceedings{solans2020poisoning,
  author = {David Solans and Battista Biggio and Carlos Castillo},
  title = {{Poisoning Attacks on Algorithmic Fairness}},
  booktitle = {ECML/PKDD},
  year = {2020},
  doi = {10.1007/978-3-030-67658-2\_10},
}
Figure