Abstract
As machine learning models are increasingly integrated into critical social decision-making processes—such as credit scoring, hiring, and recidivism prediction—ensuring their fairness has become a paramount security concern. While existing adversarial machine learning research primarily focuses on compromising model integrity (misclassification) or availability (denial of service), this paper introduces the concept of fairness poisoning. We demonstrate that a malicious actor can strategically inject a small amount of "poisoned" data into a training set to induce systematic bias against specific demographic groups. Crucially, these attacks are designed to be stealthy, maintaining high overall accuracy to avoid detection by standard monitoring tools. We present a gradient-based optimization framework for crafting these attacks and evaluate their impact on both synthetic and real-world datasets.
Introduction
The rapid adoption of algorithmic decision-making has sparked significant interest in algorithmic fairness. Most existing literature assumes that bias in models arises unintentionally from historical prejudices present in the training data. However, this paper identifies a critical security gap: the potential for intentional corruption of fairness.
An attacker may seek to manipulate a model to disadvantage a specific protected group (e.g., based on race or gender) for ideological or competitive reasons. Unlike traditional poisoning attacks that aim to degrade overall performance, a fairness poisoning attack seeks to maximize a disparity metric (like demographic parity or equalized odds) while keeping the model's error rate within acceptable bounds. This "stealthy" nature makes fairness poisoning a particularly insidious threat.
Methodology: Bilevel Optimization for Fairness Poisoning
The authors propose a novel framework for generating poisoned samples by formulating the attack as a bilevel optimization problem.
In this framework, the attacker's goal is to find a set of poisoned points $D_p$ that, when added to the clean training set $D_{tr}$, results in a model $\theta^*$ that maximizes an unfairness objective $\mathcal{L}_{atk}$ on a validation set $D_{val}$.
- Inner Loop: The model parameters $\theta^*$ are obtained by minimizing the training loss on the poisoned dataset: $\theta^* = \arg\min_\theta \mathcal{L}(D_{tr} \cup D_p, \theta)$.
- Outer Loop: The attacker optimizes the features of the poisoned points to maximize the unfairness: $\max_{D_p} \mathcal{L}_{atk}(D_{val}, \theta^*)$.
To solve this efficiently, the authors employ a gradient-based approach. By leveraging the Karush-Kuhn-Tucker (KKT) conditions of the inner optimization, they can compute the gradient of the optimal model parameters $\theta^*$ with respect to the poisoned inputs $D_p$. This allows the attacker to iteratively "nudge" the poisoned samples in the feature space to maximize the target disparity.
The attack can be executed in a white-box setting, where the attacker has full knowledge of the training pipeline, or a black-box setting, where the attacker optimizes against a surrogate model and relies on the transferability of the poisoned samples to the target model.
Experimental Setup
The effectiveness of the attack was evaluated using two primary datasets:
- Synthetic Data: A 2,000-sample dataset with two-dimensional features and a binary sensitive attribute. This controlled environment allowed for precise measurement of how initial group separation affects attack success.
- COMPAS Dataset: A standard benchmark in fairness research containing criminal recidivism data. It includes 18 features for over 6,000 offenders, with race and gender serving as sensitive attributes.
The study tested several common classifiers, including Logistic Regression, Support Vector Machines (SVMs), Naive Bayes, Decision Trees, and Random Forests.
Results and Analysis
The experiments yielded several key insights into the vulnerability of machine learning models to fairness poisoning.
Impact on Decision Boundaries
On synthetic data, the gradient-based attack successfully shifted decision boundaries to increase group disparity. As shown in the comparison below, the injection of poisoned points (represented by stars) forces the model to adopt a boundary that disproportionately misclassifies members of the unprivileged group compared to the privileged group, even when the overall accuracy remains relatively stable.
Stealth and Transferability
A major finding is that fairness can be significantly compromised with only a moderate impact on accuracy. This confirms the attacker's ability to remain stealthy. Furthermore, the attacks demonstrated high transferability; poisoned samples generated for a Logistic Regression model were often effective at inducing bias in an SVM model. This suggests that attackers do not need perfect knowledge of the target system to be successful.
Model Resilience
While linear models like Logistic Regression and SVMs were highly susceptible, non-linear models like Decision Trees and Random Forests exhibited greater resilience. However, they were not entirely immune, and the authors suggest that more sophisticated attack strategies might still be effective against them.
Conclusion and Future Work
This research highlights a new and critical threat vector in adversarial machine learning. Fairness is not just a statistical property to be monitored; it is a security property that must be defended. The success of these poisoning attacks, even in black-box scenarios, underscores the need for robust training algorithms and data-sanitization filters.
Future work should explore more complex constraints on poisoned samples to evade outlier detection and investigate the impact of these attacks on deep neural networks. Additionally, developing defenses that can distinguish between natural data shifts and intentional fairness poisoning is a vital next step for the community.
Summary updated