Perturbation Effects on Accuracy and Fairness among Similar Individuals

Xuran Li, Peng Wu, Yanting Chen, Xingjun Ma, Zhen Zhang, Kaixiang Dong, Salim, Flora D.

Abstract

Deep neural networks (DNNs) are vulnerable to adversarial perturbations that degrade both predictive accuracy and individual fairness, posing critical risks in high-stakes online decision-making. The relationship between these two dimensions of robustness remains poorly understood. To bridge this gap, we introduce robust individual fairness (RIF), which requires that similar individuals receive predictions consistent with the same ground truth even under adversarial manipulation. To evaluate and expose violations of RIF, we propose RIFair, an attack framework that applies identical perturbations to similar individuals to induce accuracy or fairness failures. We further introduce perturbation impact index (PII) and perturbation impact direction (PID) to quantify and explain why identical perturbations produce unequal effects on individuals who should behave similarly. Experiments across diverse model architectures and real-world web datasets reveal that existing robustness metrics capture distinct and often incompatible failure modes in accuracy and fairness. We find that many online applicants are simultaneously vulnerable to multiple types of adversarial failures, and that inaccurate or unfair outcomes arise due to similar individuals share the same PID but have sharply different PIIs, leading to divergent prediction-change trajectories in which some cross decision boundaries earlier. Finally, we demonstrate that adversarial examples generated by RIFair can strategically manipulate test-set accuracy or fairness by replacing only a small subset of items, creating misleading impressions of model performance. These findings expose fundamental limitations in current robustness evaluations and highlight the need for jointly assessing accuracy and fairness under adversarial perturbations in high-stakes online decision-making.

Introduction

In high-stakes domains such as loan approvals, recidivism risk assessment, and job hiring, AI models are expected to be both accurate and fair. Traditionally, Adversarial Robustness (the ability to resist small input changes that flip predictions) and Individual Fairness (the requirement that similar individuals receive similar outcomes) have been studied in isolation. However, a model might remain accurate under perturbation while becoming unfair, or vice versa. This research introduces Robust Individual Fairness (RIF) to bridge this gap, ensuring that similar individuals receive consistent, correct predictions even under adversarial stress.

The RIFair Attack Framework

To expose vulnerabilities in RIF, the authors developed RIFair, a gradient-based attack framework. RIFair applies identical perturbations to an individual $v$ and their similar counterpart $v'$ to induce specific failure modes:

  • True Biased (TB): The original instance $v$ remains correctly classified, but the counterpart $v'$ is pushed across the decision boundary, resulting in an unfair disparity.
  • False Biased (FB): The original instance $v$ becomes incorrectly classified, while the counterpart $v'$ remains correct.
  • False Fair (FF): Both individuals are pushed across the decision boundary; while the outcome is "fair" (consistent), it is now universally incorrect.
Visualization of RIFair attack types: True Bias (TB), False Bias (FB), and False Fair (FF) attacks relative to decision boundaries.

Key Insights: The Asynchronous Flip

A central contribution of this work is explaining why these failures occur. The researchers introduced two diagnostic metrics:

  1. Perturbation Impact Direction (PID): Indicates whether a perturbation pushes a prediction toward a positive or negative outcome.
  2. Perturbation Impact Index (PII): Measures the magnitude of the prediction change (sensitivity).

The study found that similar individuals often share the same PID but possess significantly different PIIs. This leads to an "asynchronous flip": under the same adversarial "nudge," one individual hits the decision boundary and changes their outcome much sooner than their counterpart.

Prediction trajectories and perturbation impacts (PII/PID) for similar individuals v and v' under different attack scenarios.

Experimental Results

The framework was tested on seven model architectures (including BERT, Transformer, and MLP) across four datasets (Census Income, Bank Marketing, COMPAS, and ACS Employment). Key findings include:

  • Fragility of Fairness: Models were found to be significantly more vulnerable to biased predictions than to simple inaccuracies. On average, only 26.55% of instances remained both accurate and fair after perturbation.
  • Metric Misalignment: Rankings of models based on accuracy robustness do not correlate with fairness robustness. A model that is difficult to "trick" into an error may still be easily "tricked" into bias.
  • Model Performance: BERT demonstrated the highest overall resilience among the tested architectures, though it still exhibited significant vulnerabilities.
  • Strategic Manipulation: The authors demonstrated that replacing a small fraction of a test set with RIFair-generated examples can artificially manipulate reported metrics, highlighting a major flaw in current AI auditing practices.

Conclusion

The RIF framework reveals that current robustness evaluations are incomplete. Ensuring AI safety in the real world requires a joint assessment of accuracy and fairness under adversarial conditions. Future work should focus on developing robust training methods that balance the sensitivity (PII) of similar individuals, ensuring they cross decision boundaries together rather than creating unfair disparities.

Summary updated

Abstract

Deep neural networks (DNNs) are vulnerable to adversarial perturbations that degrade both predictive accuracy and individual fairness, posing critical risks in high-stakes online decision-making. The relationship between these two dimensions of robustness remains poorly understood. To bridge this gap, we introduce robust individual fairness (RIF), which requires that similar individuals receive predictions consistent with the same ground truth even under adversarial manipulation. To evaluate and expose violations of RIF, we propose RIFair, an attack framework that applies identical perturbations to similar individuals to induce accuracy or fairness failures. We further introduce perturbation impact index (PII) and perturbation impact direction (PID) to quantify and explain why identical perturbations produce unequal effects on individuals who should behave similarly. Experiments across diverse model architectures and real-world web datasets reveal that existing robustness metrics capture distinct and often incompatible failure modes in accuracy and fairness. We find that many online applicants are simultaneously vulnerable to multiple types of adversarial failures, and that inaccurate or unfair outcomes arise due to similar individuals share the same PID but have sharply different PIIs, leading to divergent prediction-change trajectories in which some cross decision boundaries earlier. Finally, we demonstrate that adversarial examples generated by RIFair can strategically manipulate test-set accuracy or fairness by replacing only a small subset of items, creating misleading impressions of model performance. These findings expose fundamental limitations in current robustness evaluations and highlight the need for jointly assessing accuracy and fairness under adversarial perturbations in high-stakes online decision-making.
BibTeX
@misc{li2024perturbation,
  author = {Xuran Li and Peng Wu and Yanting Chen and Xingjun Ma and Zhen Zhang and Kaixiang Dong and Salim, Flora D.},
  title = {{Perturbation Effects on Accuracy and Fairness among Similar Individuals}},
  year = {2024},
  eprint = {2404.01356},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.01356},
}
Figure